SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning
Authors: Jiaqi Chen
Bang Zhang
Ruotian Ma
Peisong Wang
Xiaodan Liang
Zhaopeng Tu
Xiaolong Li
Kwan-Yee K. Wong
Venue: NeurIPS 2025 poster ()
Keywords: Self-play
critic model
Reasoning
RL
Primary area: deep_learning

Abstract: Evaluating the step-by-step reliability of large language model (LLM) reasoning, such as Chain-of-Thought, remains challenging due to the difficulty and cost of obtaining high-quality step-level supervision. In this paper, we introduce Self-Play Critic (SPC), a novel approach where a critic model evolves its ability to assess reasoning steps through adversarial self-play games, eliminating the need for manual step-level annotation. SPC involves fine-tuning two copies of a base model to play two roles, namely a "sneaky generator" that deliberately produces erroneous steps designed to be difficult to detect, and a "critic" that analyzes the correctness of reasoning steps. These two models engage in an adversarial game in which the generator aims to fool the critic, while the critic model seeks to identify the generator's errors. Using reinforcement learning based on the game outcomes, the models iteratively improve; the winner of each confrontation receives a positive reward and the loser receives a negative reward, driving continuous self-evolution. Experiments on three reasoning process benchmarks (ProcessBench, PRM800K, DeltaBench) demonstrate that our SPC progressively enhances its error detection capabilities (e.g., accuracy increases from 70.8% to 77.7% on ProcessBench) and surpasses strong baselines, including distilled R1 model. Furthermore, SPC can guide the test-time search of diverse LLMs and significantly improve their mathematical reasoning performance on MATH500 and AIME2024, surpassing those guided by state-of-the-art process reward models.

SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning
Jiaqi Chen'
Bang Zhang"
Ruotian Ma
Peisong Wang
Xiaodan Liang*
Zhaopeng Tu² Xiaolong Li
Kwan-Yee K. Wong'
'The University of Hong Kong Tencent
TTsinghua University "MBZUAI
Project: https://chen-jjude 1othP/
Abstract
Evaluating the step-by- reliability of large language model (LLM) reasoning, such Chain-of Th-ough,,,, remains challenging due the difficulty and cost of obtaining high-quality step-level supervision. In this paper, we introduce Self- Play Critic (SPC), novel approach where critic model evolves its ability assess reasoning steps through adversarial self-play games, eliminating the need for manual step-level annotation. SPC involves fine-tuning copies of base model to play two roles, namely a "sneaky generator" that deliberately produces erroneous steps designed to be difficult detect, and a "critic" that analyzes the correctness of reasoning steps. These two models engage in an adversarial game in which the generator aims fool the critic, while the critic model seeks to identify the generator's 's errors. Using reinforcement learning based on the game outcomes, the models iteratively improve; the winner of each confrontation receives positive reward and the loser receives a negative reward, driving continuous self-evolution. Experiments on three reasoning process benchmarks (ProcessBench, PRM800K, DeltaBench) demonstrate that our SPC progressively enhances its error detection capabilities (e.g., accuracy increases from 70.8% to 77.7% on ProcessBench) and surpasses strong baselines, including distilled 1 model. Furthermore, SPC can guide the test-time search of diverse LLMs and significantly improve their mathematical reasoning performance on MATH500 and AIME2024, surpassing those guided by state-of-the-arr process reward models.
Introduction
The Chain-of-Thought (Co'T) [1:3] reasoning process, which emerges in the autoregressive generation of large language models (LLMs), has been applied to address a variety of complex tasks [3]
Training methods such as Supervised Fine-Tuning (SFT) [4] 13), Reinforcement Learning from
Human Feedback (RLHF) [16] [7]. and self-play reinforcement learning [8] 19. have demonstrated success in obtaining high-quality COT. Recently, the popular ol (20), RI 211, and QwQ [22] utilize large-scale reinforcement learning for training and employ test-time scaling to generate long CoT, further enhancing their reasoning capabilities. As the CoT generated by LLMs becomes increasingly complex and diverse, it is particularly important to verify the reliability of the reasoning process, analyze the potential errors in reasoning steps, and guide the test-time search improve the reasoning process 23
number of verification models have been developed analyze and evaluate the reasoning process of LLMs. For example, outcome verifiers 23 provide outcome-level validation to rank or reward multiple responses from LLMs. Process verifiers 23 25), which validate each step in the reasoning
Independent researcher. Cooresppnning authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Incorrect Steps
will sneakily add subtle errors to fool the critic.
I need to carefully figure out all the hidden mistakes.
Reward +1 for Winner
against
Reward -1 for Loser
Sneaky Generator
Step Critic
Step Critiques
Figure 1:
We continuously generate reinforcement training samples for the critic through adversarial games. The sneaky generator aims to create subtle erroneous steps to challenge the critic, while the critic must accurately distinguish between correct and incorrect steps from mixed input of them. Benefiting from the opposing optimization objectives, both models can evolutionally learn from each other, akin to how humans improve their skills in board games through competition.
However, there several challenges that limit the development of such step-level approaches. Firstly, while it is relatively simple to extract the final predicted answer and determine the correctness of a solution, determining the correctness of reasoning step and automatically obtaining well-annotated step data for training a process verifier is much more difficult. Secondly, LLMs are updated rapidly, and heavy human expert annotations on the outputs of specific LLMs may not be applicable to the latest LLMs due to distributional differences. Thirdly, the dataset limited to step correctness annotations restricts the training of critic model - preventing it from providing substantive feedback and reducing it to merely& a scoring mechanism for verification.
process, have proven crucial in recent advances in LLM reasoning 226 32 34
In this paper, introduce novel Self-Play Critic (SPC) to diagnose potential errors and provide valuable critiques for each step in the mathematical reasoning process. Inspired by the self-play framework [1]], we propose adversarial game between a sneaky generator and critic to contin- uously generate samples for reinforcement learning, thereby evolving the capabilities of the critic model. Specifically, we first employ supervised fine-tuning to initialize a base model as sneaky generator, converting correct steps into incorrect steps that can significantly impact the success rate of problem-solving. Concurrently, we initialize identical base model to play the role of critic, whose goal is to identify the correctness of these reasoning steps and provide some critiques for them. As shown Fig. [ we put these two models in an adversarial game by feeding the incorrect steps successfully generated by the sneaky generator to the critic. Through this adversarial game, we anticipate that the sneaky generator can simulate errors that can practically influence the reasoning of LLMs while remaining difficult for the critic to detect. On the other hand, the critic is expected to gradually address its shortcomings and improve its ability to catch all errors in the reasoning steps. Benefiting from this design, we continuously generate positive/negative samples from different LLMs for reinforcement learning without the need for additional human annotations, facilitating the iterative evolution of critic model which can provide valuable step critiques.
Extensive experiments have been conducted validate the effectiveness of our proposed self- play critic. After one round of supervised fine-tuning on Qwen2.5-7BB --B--Intttt and two rounds of iterative reinforcement fine-tuning, our SPC has shown continuously evolving performance on three human-annotated reasoning process assessment benchmarks (ProcessBench [27]. PRM800K (23) and DeltaBench (35). For instance, the average accuracy of SPC on PRM800K has gradually improved from 71.0% to 75.8%, surpassing the 71 1.4% performance of the same-sized distilled model of R1 (1 We further introduce a new approach utilize our tailored critic model, wherein the critic predicts the correctness of each step during LLMs' test-time search. This allows the LLM to promptly abandon incorrect steps and regenerate new steps, rather than waiting until the entire solutions are generated and then scoring them using verifiers. Experiments on MATH500 (36 and AIME2024 (7 indicate that SPC can enhance mathematical reasoning for three different types of L.LMs, including popular Llama 38), Qwen [12, and distilled RI (21 with long CoT reasoning process.

---

2
Related Work
LLM Reasoning Powerful large language models (LLMs) 4 733 are becoming increasingly adept at constructing Chain-of-Thought (CoT) to tackle complex reasoning tasks, such solving math problems and code generation. The recently popular ol [20] RI 22 and QwQ 22 models are further equipped with exceptional deep thinking capabilities, allowing them to construct long CoT during inference to decompose complex tasks and even perform extensive self-critique and self-correction [1]. However, eee-grained analyses in recent research [35] indicate that the effective proportion of self-critique in these long CoT is still very low, and biases exist in the self- critique of their reasoning processes. It is therefore necessary have simple external critic for assessing the reasoning steps of various L.Mss, providing step-level critiques.
Verification and Critique for LLM Verifiers 23 can enhance reasoning performance by ranking or integrating multiple responses generated by LLMs during inference. Additionally, they can also provide more accurate rewards during training to guide the optimization of L.S.s. Verifiers can be categorized into two types, namely outcome reward models (ORMs) and process reward models (PRMs). ORMs provide solution- level scores for the entire problem-solving process, whereas PRMs assign step-level scores to each reasoning step, which can be aggregated to produce more accurate solution-level score. Recent works (28 30 (2 propose critic models for verification, arguing that scalar scores have limited ability in evaluating the outputs of LLMs. In contrast, feedback in natural language form can activate the thinking capabilities of LLMs, resulting in more reliable critiques to represent the correctness of reasoning. In this work, explore how to analyze the correctness of the current step and provide step-level critiques based on partial reasoning steps.
Self-Play Self-play 43 is a method in reinforcement learning where an agent interacts with several copies of itself in an environment to learn specific actions. A significant advancement in self- play is demonstrated by AlphaGo 45] and AlphaZero [46) which greatly surpass human champions in the game of Go, without the need for human knowledge in training. Recent studies apply self-play to L.LM alignment and enhancement solution-level game between a powerful prover and a weak scoring verifier to enhance the legibility of the LLM, though resulting in performance degradation. Cheng et al. [47] introduced Taboo language game between attacker and defender to improve LLMs' reasoning abilities. In this paper, we design an adversarial game to generate data for training a step-level critic, which provides correctness analysis for the reasoning steps of LLMs.
50). For example, Kirchner et al. [19] proposed a
3 Methodology
3.1
Overview
Training a step-level critic requires large amount of data annotated with step correctness. However, collecting step-level data presents considerable challenges. First, identifying and annotating the reasoning errors of powerful LLMs requires professionals with relevant expertise. Second. LLMs are rapidly updated, and the labor-intensive annotations may become outdated and inapplicable the latest L.LMs due to distributional shifts. Third, there is no definite answer for each step, complicating the definition of "incorrect" and the automation of the annotation process.
In this work, we design a self-play framework to enable the self-evolution of step-level critic by automatically producing step-level annotation through adversarial game. As shown in Fig. 2 our framework involves two opposing models, i.e., a sneaky generator and step-level critic C.
Sneaky generator converts correct reasoning steps from LLMs into incorrect ones, automating the creation of numerous steps with potential errors. Its goal is generate sneaky steps that not only decrease the reasoning success rate of LLMs but also deceive the critic (i.e.. the critic fails to detect the erroneous steps). Specifically, given a problem p and a correct partial reasoning trajectory
T.k = (41, (2,...x)) produced by an L.M solver, the sneaky generator S converts the last correct step 1 into a candidate sneaky step ti. = Sp, T, (2). This candidate becomes valid sneaky step 13 if it significantly impacts the solver's success rate in the subsequent completion from this step. If the sneaky step is invalid or the critic detects the errors in valid sneaky step, the sneaky generator receives negative reward. Conversely, if the critic fails to detect the errors in valid
3

---

Problem:
Step Critic
Convert S1011001_ 1_2 base
Reward +1
Partial Solution:
convert binary number
W(1011001 20) to base we can
Incorrect Step: 3. "Combine digils" An Combining these base digits, we obtain:'n W/nn 1011001_ _ 11121_4hn N)
Critique:
follow these steps:
The last step involves combining the digits obtained from the conversion of each
1. ""Group the binary digits into pairs starling from the right"":
binary pair ... The correct combination should be W11121 _.), but the solution incorrectly writes it as 1i(1 1(11121_4))
Since each base digit can be represented by 2 birary digits...
Sneaky CoT: ... use "Enror Type
calculasion erro"", SpeciTically, I will introduce an incorrect simplification in the concatenation of the base 4 digits...
Conclusion:
The correctness of the last
Correct Step: "Combine base 4 "ig Combining the base 4 digits from the pairs, we gettin Vinn 1011001 2 1121 1121_4nn
Sneaky Generator
Reward -1
Figure 2: The framework of proposed SPC. We randomly select correct step along with the partial solution before that step and feed them into the sneaky generator, which first selects one of the predefined error types and then converts the correct step into an incorrect step. The successfully generated incorrect step is then fed to the critic for eπтоr detection. If the critic successfully identifies the error, it receives reward of +1, while the sneaky generator incurs a reward of 11 If the critic is deceived, the critic and sneaky generator are rewarded -1 and +1, respectively.
sneaky step, the sneaky generator receives a positive reward. By automatically evaluating the success of the generated sneaky steps, then employ reinforcement learning enable the self-evolution of the sneaky generator.
Step Critic aims to identify all potential errors in the reasoning steps of LLMs. In each iteration of the adversarial game, the critic's role is to detect all error steps generated by the sneaky generator. Specifically, given a partial reasoning trajectory T = (t1, .t,..)))) and a valid sneaky step t produced by the sneaky generator S, the critic C is expected identify t by generating a step-level critique. The success or failure of detecting the sneaky step determines the critic's 's rewards, allowing continuous optimization through reinforcement learning.
Overall, these two models have opposing objectives, allowing them to evolve through adversarial self-play. In the following sections, explain how initialize these models and continuously generate positive and negative samples for reinforcement learning through adversarial games.
3.2 Initializing Sneaky Generator
To initialize the sneaky generator So. we train the base model Qwwnnt.5-7B-Instruuc [12 using Supervised Fine-Tuning (SFT) to equip it with the fundamental capability to generate incorrect steps. To ensure the accuracy of the initialization data, we use correct-incorreet step pairs from PRM800K (23] to construct an eтror step transformation process. Specifically, we extract correct- incorrect step pairs < fi. t > with the same problem p and partial solution Tik-1 (t1, 12tx-)) (the steps preceding the extracted pairs) from PRM800K. We next prompt GPT-4 to create a chain- of of-thought transformation (cor(t)) + t by first selecting an error type from five predefined common еттor types (see Sec. [ and then performing a detailed transformation. This process results in transformation behavior cloning dataset (x,y) D where x (p, 1. t() as input, y ( - E as output. We then finetune Owwen..57BB-Instte policy To for the initial sneaky generator Sp using the SFT loss:
on dataset DE obtain
LSFT = -B(x.)>Dg [log (y]x)]
Automated Validation for Sneaky Generator To form an adversarial game, we need to annotate the generated steps and feed actual incorrect steps to the critic model. However, existing L.MM- as-a-Judge methods [35 51 inevitably introduce bias, while the manual annotation is excessively labor-intensive. We therefore propose evaluating the impact of different steps on the problem-solving success rate to ascertain whether sneaky step can be considered incorrect. Concretely, based

---

correct solution generated by an open-source LLM, we first sample an original step and transform it into a sneaky step using the sneaky generator. We subsequently use the same LLM to complete the entire reasoning process after the original/sneaky steps, and this is repeated N times. If the original step achieves relatively high success rate while the sneaky step results in a significantly lower success rate, we consider this pair of steps to represent correct and incorrect steps, respectively. In our experiment, we adopt a strict criterion to ensure data quality. If the original step achieves a success rate greater than equal to 75%, while the sneaky step results in success rate of 0%, we then collect this pair of steps for subsequent adversarial games.
3.3 Initializing Step Critic
Based on the results from ProcessBench 27), reasoning models such as QwQ [22 and distilled RI models [21] outperform non-reasoning models such as GPT when serving as critic models. However, the lengthy reasoning process in R1 leads to slow and redundant model generation, and its followiinnn capability relatively poor, often failing to produce a concise critique with definite conclusion about the correctness of step. We therefore combine the strengths of both types of models when initializing the critic.
Specifically, we prompt DeepSeek-R1 R1-Distill Distill-Qwen-77 Qwen-7B as critic, taking problem p, partial solutions T.k-1, and mixed correct and incorrect steps t3 from PRM800K dataset as inputs, collect long critiques. We then employ GPT-4 to rewrite them into brief standardized critiques Qe (see Sec. [). Concretely, Q: contains an analysis of the partial solution and the current last step, as well definite conclusion regarding the correctness of the step. This also simplifies the task and facilitates the use of SFT for policy initialization. Additionally, when preparing the training data for the critic, we mix the steps labeled as correct and incorrect in PRM800K at a 1:1 ratio to ensure the critic's capabilities are balanced. We utilize human annotations from PRM800K to filter around 21,8K correctly generated critiques Qi- Similarly, we prepare behavior cloning dataset (x, y) D for the critic, where x = (p, 1 ) as input, and y = Qt as output. We then finetune the base model using SFT loss (1) to obtain an initial policy Co for the critic.
3.4 Adversarial Game
We further reinforce the models' correct behavior and continuously improve their performance, avoiding the limitations related to the scale and distribution of by recent self-play practices 19 47), we propose a step-level adversarial game between the sneaky generator and step critic, enabling continuous reward generation and self-evolution of the two roles.
PRM800K. Inspired
In each iteration of the adversarial game, we begin by using LLM solvers to generate of original step-by-step solutions for each problem. To ensure data diversity, we employ various LLM solvers from different model families, with sizes ranging from 7B to 32B, thereby enriching the diversity of sample styles. We then design an adversarial game for the two roles based on these solutions. Single steps are randomly selected from solutions for sneaky transformation, and the incorrect steps successfully produced by the sneaky generator are then fed into the critic to generate critiques. In addition ensuring that the generated step contains an error, we expect the sneaky generator to generate incorrect steps with subtle flaws that can fool and challenge the critic. Meanwhile, the critic should be powerful enough avoid being misled by any errors and provide an accurate critique.
In this game, we can set the rewards for the sneaky generator and the critic respectively in an adversarial instance as follows:
1, Sneaky Generator Wins
Raneaky
(2)
-1, Sneaky Generator Loses
Critic Wins
Reritic
Critic Loses
This opposing optimization goal enables both the sneaky generator and the critic to continuously improve their performance, achieving iterative self-evolution.
5

---

3.5 Evolving via Reinforcement Learning
In each iteration, after obtaining positive and negative samples through the adversarial games, we apply offline reinforcement learning to the critic and sneaky generator, respectively, enabling self- improvement of both roles based on the game result. Specifically, we adopt the following optimization objective to achieve stable RI. training:
V voL(o) = -Ex~D, yy-.ully-myyyl)))
Te (y xx.
ATM (x Vo log No (y [x)]
(4)
Mold
where Vold denotes the policy used to collect the offline dataset, mo (y(x)) is the importance ratio, [xa
and ATa represents the advantage estimation. Inspired by recent RLOO 52 and GRPO [3 we
formulate A [(x, y) - [KL[ma|||Taa where baseline b (the average reward of all samples) subtracted for advantage estimation, and a Kullback-Leibler (KL) penalty is added to regularize the policy тh and prevent it from deviating too far from the initial policy Tee
For the sneaky generator, considering that we also need it to generate actual incorrect steps, we treat sneaky steps that fail to affect the problem-sol saie-solliiggg wing success rate as negative samples. Additionally, sneaky steps that successfully impact the L.M success rate but do not deceive the critic will also be considered negative samples. Meanwhile, the ones that can both influence the LLM success rate and deceive the critic are considered positive samples. Consequently, our data for training the sneaky generator includes 1:1:1 ratio of positive samples and two types of negative samples.
As for the critic, we mix some correct steps from correct solutions with some incorrect steps generated by the sneaky generator for the critic to predict. The samples that the critic successfully predicts receive a positive reward, while those that are incorrectly predicted receive a negative reward. Ultimately, positive/neegatie samples each constitute half of the total samples.
Based on the adversarial game, we apply iterative training to enable continuous evolution of the two roles. Specifically, in each iteration, the newly updated policies re-engage in the adversarial game to generate new data for training, thereby evolving themselves further. Additionally, an interesting phenomenon that more balanced adversarial games contribute to the self-evolution of models. In fact, the initial sneaky generator So is weaker than the initial critic Co, resulting in an unbalanced win rate. Moreover, S obtained through synchronous iteration is even weaker than G. Therefore, we adopt an asymmetric evolution strategy, where Si competes against Co in more balanced game to generate the second round of data. This enables C2 trained in the second round to further improve its performance. Such a strategy analogous humans preferring to improve their skills in chess by playing against equally matched opponents. We provide more detailed analyses of the evolving strategies in Sec. 4.3
observe
3.6 Enhancing LLM Reasoning
Previous process reward models (PRMs) | 27 require scoring each step of the fully generated solutions and then integrate all the scores. However, after the first reasoning step error occurs, the ILMM should promptly correct the mistake. Continuing to generate more potentially flawed reasoning steps after an erroneous step is unnecessary, and the scores produced are unreliable. contrast, we propose new approach that directly employs critic to assist the LLM in searching for reasoning steps. During testing, we use "An'n' to control the LLM to output one step at a time, allowing the critic to verify the correctness of each step. If the step is correct, the search continues; if incorrect, the LLM is required to regenerate the step (up to five attempts before skipping). Our SPC effectively enhances the reasoning performance of the LLM using this approach.
4 Experiments
4.1 Experimental Settings
Evaluation We adopt PRM800K 23 ProcessBench [27], and DeltaBench (35] that include human annotations of mathematical reasoning steps for evaluation. The original setting of ProcessBench and DeltaBench is to identify the position of the first or all errors in complete solution. We argue that, in practical scenarios, critic can enhance reasoning performance by identifying the incorrect

---

Table 1: Comparison of recall on ProcessBench. We evaluate different models on their ability to assess the correctness of the current step, instead of only predicting the index of the first error in the complete solution. 'Round O' refers the initialized critic model.
Olympiad- Bench
Omni- MATH
Models
GSM8K
MATH
Average
Process Reward Models (PRMs)
Math Sww--hhd-eppeee PRM 26
58.0
58.4
68.0
64.1
62.1
27
77.0
72.9
66.9
62.1
69.7
Prompting LLMs as Critic Models
Llama-3.1 Instruct 10
59.5
57.7
53.6
53.9
56.2
Llama-3 1------ntttcttt [0
67.2
62.8
61.7
61.9
63.4
Swen2.5-7B-Instmuu 12
64.2
64.0
62.1
60.8
62.8
Qwen2.5- 100d255533--Inttree
76.2
68.1
63.9
69.3
GPT-40 6
75.5
70.5
64.5
70.1
DeepScck-RI-Disill--Ne
21
79.0
81.3
73.4
67.3
75.2
Our Critic Models
SPC (Round 0)
78.0
74.1
67.8
63.2
70.8
SPC (Round 1)
82,0
80.3
74.8
70,3
76.8
SPC. (Round 2)
84.2
80.8
76.5
69.2
77.7
step and requiring the LLM to regenerate, with no need to wait for completing all the steps. We therefore extract 1:1 ratio of correct and erroneous steps from each benchmark, only retain the reasoning process before these steps as a partial solution, and discard the reasoning steps after these steps. Besides, we evaluate the effectiveness of the critic models in assisting LLMs to solve math problems on MATH500 [44] and AIME2024 37] More evaluation details are provided in Sec. C
we primarily evaluate two types of baselines, namely Process Reward Models (PRMs) and prompting LLMs critic models. For PRMs. we select two representative methods, namely Math-Shi aath-Shepherd [26] and Qwen2.55Mathh 7--PRM000K [27]. Math- Shepherd trains process reward model through an automated data annotation process and can be utilized rank multiple outputs or ensemble them to enhance reasoning performance. Qwen2.5- Math-7B-PRM80N is based on the advanced math-specialized model Qwen2.5 Qwen22.5MMath77B [55] and further fine-tuned with the PRM800K dataset, obtaining state-of-the-ar performance among PRMs. We also prompt multiple types of LLMs to serve as critic models, using the same prompts as our SPC. Several representative models, including Llama [0], Qwen [12. RI [21, and GPT-4o [6], are selected as baselines.
Baselines Following ProcessBench 277
4.2
Main Results
Critic Performance on Reasoning Process Benchmarks As shown in Tabs. and 2 our critic models with other baselines on 3 math-related reasoning process benchmarks to evaluate the abilities of predicting step correctness. We can observe that: (1) Our proposed SPC is gradually evolving and achieves state-of- offte--f----rttt performance among all models. For example, the average performance on ProcessBench has improved from 70.8% to 77.7%, and on DeltaBench from 54.9 44.9% to 60.5%. (2) On all benchmarks, our method outperforms the latest PRMs specifically designed scoring steps. (3) The performance of prompting LLMs as critics is not as good as SPC. Our method outperforms the distilled R1 model with the same size of 7B parameters. (4) Some baselines (PRMs and prompting Llama) have imbalanced recall between correct and error steps, leading to poor harmonic mean, whereas critic is more balanced. (5) Our critic is trained short CoT data (from Qwen and Llama), but is able to generalize to long CoT reasoning steps (c.g., RI 21] and QwQ 22 in DeltaBench. In contrast, the two PRMs trained short CoT show significant performance decline in DeltaBench, with HarMean scores only 14.3% and 41.3%, respectively.
we compare
The Effectiveness of Guiding Test-Time Search Existing PRMs can enhance performance by ranking the completely generated reasoning steps or by aggregating scores using self-consistency [2] 26.. We apply the proposed SPC to LLM reasoning search, utilizing SPC to check the correctness of each step and regenerating the step it incorrect (up retries). Moreover, SPC can be combined with self-consistency conducting majority vote over several independent searches. For fair
7

---

Table 2: Comparison of our SPC with baselines on the test set PRM800K [23 and DeltaBench [35. where human-anmoten annotated correct and erroneous steps are extracted to evaluate the recall of critiques. "Correct" and "Error" represent the recall on correct and erroneous steps, respectively. "Average" denotes their arithmetic average and "HarMean" refers to their harmonic mean.
PRM800K
DeltaBench
Models
Average HarMean Correct Error Average HarMean Correct Error
Process Reward Models (PRMs)
Saah-Shepheed-PRM-- --t.h--------PRRR
26
50.0
49.5
55.2
44.8
53.3
14.3
7.69
98.8
27
73.6
73.6
74.4
72.8
58.5
41.3
90.1
26.8
Prompting LLMs as Critic Models
Llama-3. 3.1-B- -SS-Instruct
[0
51.9
30.5
18.6
85.2
49.1
6.38
3.30
95.0
Llama-3.1 3.--a---.--0-------- 1-70B-Instruct
10
54.6
38.9
25.3
83.9
44.6
20.3
11.7
77.5
Owen2.5-7B-instruc [12
52.8
37.2
24.1
81.6
48.2
33.8
21.8
74.7
12
59.0
50.5
36.6
81.4
44.7
33.0
21.8
67.6
GPT-4o 6
68.5
68.4
70.3
66.6
49.9
48.7
42.0
57.9
DeepSeek-Re RI-K--R---DR RI--Distill-aee RI-Distill-Qwenn-11 21
71.4
71.2
67.3
75.5
50.9
50.6
$4.9
46.9
Our Critic Models
SPC (Round 0)
71.0
70.8
67.8
74.2
54.9
53.5
45.9
64.0
SPC (Round 1)
72.8
70.3
59.4
86.1
58.8
57.3
68.4
49.3
SPC (Round 2)
75,8
75.8
74.8
76.9
60.5
59.5
68.2
52.8
77.5
76.8
Whnning Drikk
Wanning Cri--
Atracking So ver owo towa from Soiver
75.0
Winning Citlc lei Rowars from Sohver) Winring Officc (ell Rewarl from Sowver
75.0
72.0
70.8
8
b0.3
71.6
21.520.6
CRti
67.5
65.0
13.2
02.8
8.7
Baso
Self-Play
Self-Play
Figure 3: Ablation study our critic and sneaky generator. Left: The impact of different strategies on evolving critic models. Right: The success rate of sneaky generator in attacking LLM solver and its win rate against round and round critics.
comparison, all methods incorporating self-consistency sample 5 outputs in our experiments. In addition, for experiments without using self consistencs, we run them at least three times and average the results to reduce randomness. As shown in Tab. B two popular benchmarks MATH500 [54] and AIME2024 [37], SPC significantly improves the performance of three types of LLM solvers, and outperforms five baseline verifiers. For instance, using the Qwen Solver AIME2024, SPC combined with Celf-Consistency achieves a problem-solving accuracy of 23.3%, which is superior S.wen2.5-Math-77-PPRM-S
to the 16.77 16.7% accuracy of Self-Consistency trained using only short CoT data, yet it can still generalize to the DeepSeek-PR RI-Distill-Qwent model, which outputs in long CoT style. achieves 94.0% accuracy MATH500, whereas Math-Shepherd and
Notably, SPC is
achieve only 89.2% and 91.8%, respectively.
4.3 Ablation Study
The Impact of Different Strategies on Evolving Critic In Fig. 3 (left), we test critic models on ProcessBench, demonstrating the impact of different adversarial training methods. We refer to the sneaky generator and critic initialized after SFT as Sneaky-O and Critic-O, respectively, while Sneaky-n and Critic-n represent models trained with n rounds of self-play adversarial data. In round 1, Sneaky-1 and Critie-I are trained using data generated from the adversarial game between

---

Table 3: Performance of various methods for assisting different LLMs in math reasoning. By integrating Ceff-Consistenyy with our SPC, we achieve the best results across three types LLMs on MATH500 and AIME2024 datasets.
Solvers
Verifiers
MATH500
AIME2024
w/o
47.0
4.27
Self-Consistency
55.6
3,33
Math-Shephere
26
52.4
3,33
Wwen225-Math-7-PPRRM000
27
54.6
3.33
1aam-3.1-88-Inan
10
Self- Self-Consistency + Math-Shepherd
53.6
6.67
Self Cols-ossistnncy Qwen2.5-Main Mathh---attBB--- Ma-tthh---PRM--- PRMMO0KK
60.4
3.33
SPC (Ours)
54.5
5,63
Self-Consistency + SPC (Ours)
62.8
6.67
w/o
78.0
14.4
-ell-Consistency
82.0
16.7
Math-Shepher
78.8
13.3
82.8
16.7
Owenn2.5322-Innsti 12
Self- Self-Consistency + Math-Shepherd
80.8
13.3
Self- Self-Consistency Qwen2.5 1--------------MM---
84.6
16.7
SPC (Ours)
83.0
17.7
Self-Consistency- SPC (Ours)
85.2
23.3
87.7
53.8
Self-Consistency
92.2
70.0
Math-Shepherd
87.0
53.3
84.2
63.3
DeepScck-RII ---ick-ll--i--llll--- -istiill-Qenn-7B
Self-Consistency Math-Shepherd
89.2
60.0
Self-Consistency Owwen.5-MMah-775 PB-MMBBUKK
91.8
73.3
SPC (Ours)
92.3
52.6
Self-Consistency SPC (Ours)
94.0
73.3
Sneaky-O and Critic-O. For each successfully transformed erroneous step, we have the critic predict four critiques, which may include both correct and incorrect predictions, forming pair of positive and negative samples with the same input but different outputs. This method of constructing paired samples more effective in RL training, improving the critic from 70.8% in round 76.8%. whereas not constructing paired samples only achieves a performance of 75.0%.
For round 2, explore evolving strategies. (1) Generating round 2 data using the confrontation between Sneaky-1 and Critic-1 and mixing it with the data from round 1. We observe a significant 72.0%, possibly
performance decline in the critic trained with this setting, dropping from 76.8% due to overfitting. We notice that the win rate of Sneaky-1 against Critic-1 is only 13.2%. Therefore, such an overly unbalanced game might prevent the critic from learning new knowledge from the adversarial process, similar to how humans need opponents of comparable skill levels when playing chess. Therefore, we adopt another setting: (2) Generating data through the game between Sneaky- and Critic-O, given that Sneaky-1 had win rate of 32.5% against Critic-O. We then mix the data from both rounds for training Critic-O and update it as Critic-2. Balancing the game prevents performance degradation and enables self-evolution, improving SPC's performance to 77.7%.
The Performance of Sneaky Generator As shown in Fig. (right), we analyze sneaky generators' success rates attacking 2-we-225-7B-Inntteee solver, as well as their win rates against Critic-O and Critic-1. It is observed that the proportion of successful attacks on the solver gradually increases from 21.5% to 33.6%, as the sneaky generator iterates. We then feed successfully generated erroncous steps to the critic models. Sneaky generators' win rates against Critic-0 increase from 20.6% (Sneaky-O) to 30.3% (Sneaky-2). Overall, the performance the sneaky generators iteratively improved.
We also analyze a training setting without adding failed attacks on the solver negative samples, using only successfully generated erroneous steps to construct positive/negative samples for training Sneaky-1, referred to as "w/o Reward from Solver" with lighter colors. We find that this approach severely impacts the performance
the sneaky generator, significantly reducing the proportion of successful attacks to 12. %. Among the successfully attacked samples, the proportion that could deceive the critic also very low, achieving a 19.6% win rate against Critie-O. Therefore, crucial to ensure that the sneaky generator receives rewards from both the solver and the critic.

---

Conclusion
In this paper, we propose a self-play critic with the ability of detecting step-level LLMs reasoning errors. Specifically, we design a sneaky generator to produce incorrect steps and critic to assess the correctness of each step. Through the adversarial game between these two models, we can continuously generate positive and negative samples for reinforcement learning. The results three reasoning process evaluation benchmarks fully demonstrate the effectiveness our SPC. Furthermore, we apply SPC to assist LLMs' test-time search, further enhancing their reasoning performance.
References
[1] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 15:24824-248877 2022.
[2] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint ar arXiv:2203.11177 2022.
[3] Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, et al. Thoughts are all over the place: On the underthinking of ol-like 1lms. arXiv preprint arXiv:2501. 18585, 2025.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances Neural Information Processing Systems, volume 33, 2020.
[5] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2333..88774 2023.
[6] Aaron Hurst. Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. 2024.
[7] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean Saan-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: A family of highly capable multimodal models, 2024.
[8] Anthropic. Introducing the next generation of claude, 2024. URL httpps//w// httttp::/wwwantthoop... com hom//ews//claude 3--iauuee--family
[9] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[10] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXivv2407.2111 2024.
[11] Qwen Team. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2024.
[12] Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https:// gvenlm.github.. wtt.ii/ooooooeeeeeeooeoo
[13] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arX1v:2412.19437, 2024.
10

---

[14] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[15] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instructionffinttune language models. arXiv preprint arXiv: 10. 1416, 2022.
[16] Daniel M Ziegler, Nisan Stiemnon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: (0999999999999 2019.
[17] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022.
[18) Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint Mxiv:24001.0333. 2024.
[19] Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda. Prover-verifier games improve legibility of Ilm outputs. arXiv preprint arXiv:2407.13692, 2024.
[20] OpenAI. Openai ol system card. preprint, 2024.
[21] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
[22] Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, November 2024. URL https://quenlnn ....mngiiig github..... 1o. wwt.:lg/////////oo http//qqq3330 gwq-32b-am preview/
[23] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL https
[24] Skywork ol Team. Skywork-ol open series. ber 2024. URL htR.:tttt:::uuuggigaaaeee co/Skykork
Novem-
[25] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback, 2022.
[26] Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. shephshhpherd: Verify and reinforce llms step-by-step without human annotations, 2024.
[27] Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Processbench: Identifying process errors in mathematical reasoning. arXiv preprint 11X.v2241..0655, 2024.
[28] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. The 4th Workshop on Mathematical Reasoning and Af at 24,,, 2024. URL https: itttps://operreveww net/
#orum?id-CxHHRoTLmYYY
[29] Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, et al. Self-generated critiques boost reward modeling for language models. arXiv preprint arXiv:2411.1666 2024.
11

---

[30] Xin Zheng, Jie Lou, Boxi Cao, Xueru Wen, Yuqiu Ji. Hongyu Lin, Yaojie Lu, Xianpei Han, Debing Zhang, and Le Sun. Critic-cot: Boosting the reasoning abilities of large language model via Shain-of-thoughis critic. arXiv preprint arXiv:2408. 16326, 2024.
[31] Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. Sar: Teaching llms to self-verify and self-correct via reinforcement learning. arXiv preprint arXiv:2502. 1283,, 2025.
[32] Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reasoning makes smaller Ilms stronger solvers. arXiv preprint arXiv:2008.06195, 2024.
[33] Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. To- ward self-improvement of Ilms via imagination, searching, and criticizing. arXiv preprint arXiv:2404.1 1205: 2024.
[34] Zhen Huang, Haoyang Zou, Xuefeng Li. Yixiu Liu, Yuxiang Zheng. Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. O1 replication journey - part 2: Surpassing ol-preview through simple distillation big progress bitter lesson? Github, 2024. URL
https://githut:
GATR-MLPPO1--Journs
[35] Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Zhongyuan Peng. Zhaoxiang Zhang, Zhicheng Zheng, Wenbo Su, and Bo Zheng. Can large language models detect emrors in long chain-of- thought reasoning?, 2025.
[36] Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. Measuring faithfulness in chain-of-thoughh reasoning. arXiv preprint arXIvv23007133702, 2023.
[37]
AI-MO. Aime 2024, 2024. URL
AIME 2024
[38]
Meta. Introducing meta Ilama3: The most capable openly available Ilm to date, 2024. URL https://ai. meta. wow/bloo/////oooooocoooooooooooo
httm:/loogmmtaaalaaaaa mwtammmtaalie 11ama-3/
[39] Hugging Face, Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https://gtt http//gtttttttt///g////e com / hww.hhhggggggg/////gg/gggggg auggingfaaceoppe--r11 open-r1
[40] Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing Hc, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. https:://kksst: pllp.ooioon.. wite/simpeerrii reason 2025. Notion Blog.
[41] Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. https//github.com/llaiiiia Tanyeeeerooooooo..oo 2025. Accessed: 2025-01-24.
[42] Ruihan Yang, Fanghua Ye, Jian Li, Siyu Yuan, Yikai Zhang, Zhaopeng Tu, Xiaolong Li, and Deqing Yang. The lighthouse of language: Enhancing Ilm agents via critique-guided improvement. arXiv preprint arXiv:2503. 16024, 2025.
[43] Arthur L Samuel. Some studies in machine learning using the game of checkers. IBM Journal of research and development, 3(3):210-229, 1959.
[44] Gerald Tesauro al. Temporal difference learning and id-gammon. Communications of the ACM, 38(3):58-68, 1995.
[45] David Silver, Thomas Hubert. Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement leaming algorithm. arXiv preprint 112200171..00/15.. 2017.
[46] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. nature, 5507766)))33443399 2017.
12

---

[47] Pengyu Cheng, Tianhao Hu, Han Xu, Zhisong Zhang, Yong Dai, Lei Han, Xiaolong Li, al. Self-playing adversarial language game enhances Ilm reasoning. Advances in Neural Information Processing Systems, 37:126515-126543 2025.
[48] Ziyu Ye, Rishabh Agarwal, Tianqi Liu, Rishabh Joshi, Sarmishta Velury, Quoe V Le, Qijun Tan, and Yuan Liu. Evolving alignment via asymmetric self- arXiv preprint 1XX::2441.00000 2024.
[49] Keming Lu, Bowen Yu, Chang Zhou, and Jingren Zhou. Large language models super- positions of all characters: Attaining arbitrary role-play via self-alignment. arXiv preprint arXiv:2401.1244 124744 2024.
[50] Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675, 2024.
[51] Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, al. Justice or prejudice? quantifying biases in Ilm-as-a-judge. arXiv preprint arXiv:2410.02736 2024.
[52] Arash Ahmadian, Chris Cremer. Matthias Gallé, Marzich Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint 2404 2024.
[53] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. 2024.
[54] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint urXiv:2 2021.
[55]
Qwen. Qwen2.5-math-7b, 2024.
URL http:://buginngfccc.co// cttp:/g.....cccc.aaaoooaeeeeeoe /wen2
5- --Mtth-BB
[56] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
[57] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-leva bilingual multimodal scientific problems. arXiv preprint arXi::2402.140000 2024.
(58) Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv: arXiv:2410.07988, 2024.
13

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Sec. A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification: Sec. C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
we upload the data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuia wwwiicccguie//eeeeeeeeeis CoodeSubeiiie ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Sec. C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: we averaged the results with high randomness.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Sec. C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: hwt.a//wwwp...coooooccccccccaaaa
Answer:
[Yes]
Justification: have reviewed this.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Sec. A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Sec. C
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
18

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
19

---

Justification:
Sec. 3
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy htteas////eerrpppoccc///o/rreerp for what should or should not be described.
20

---

Appendices
A Limitations and Societal Impact
Our SPC continuously generates data for reinforcement learning through adversarial games and has achieved impressive results. However, our current experiments are limited to several representative mathematical reasoning tasks. future work, we plan extend our approach more general domains to further demonstrate the potential of the proposed framework.
Potential negative societal impacts of this work may include the misuse of sneaky generator. For example, training a general sneaky generator to produce false and misleading information. On the other hand, enhancing the robustness of LLMs against attacks and training a general critic to automate the review of false information on the internet are also worthwhile research directions.
B Prompts
We demonstrate the prompts for generating data initialize the sneaky generator and critic model.
Fig. shows the prompts for querying GPT-4 (gpt-4-turbo-2022) -urroooo222-000099 to obtain sneaky transforma- tions. These prompts include the five predefined error types, as well as correct/incorrect step pairs extracted from PRM800K. GPT-4 needs first select a corresponding error type and then output the transformation process.
Figs. 5 andd iilustrate the prompts we used to collect step-level critiques for initializing the critic model. We first use the prompts in Fig. to feed data from PRM800K into DeepSeek-RI-DDe -D-istlll-- Qwen-7B, collecting a batch of raw critiques. However, these critiques often do not follow our instructions in standard format, making it difficult for us to assess the correctness of the critiques. Additionally. the responses are often too lengthy and include lot of reflection and exploration, which is not conducive to performing SFT on the base model. Therefore, we feed these raw critiques for refinement, as shown in Fig. 66] By leveraging GPT-4o's strong instraction-following capabilities, we summarize the unstructured eritiques into a concise and standardized version for subsequent SFT training.
(referred to as draft critiques in the prompt) into GPT-4o (gpt-40-2024-08-
When actually training the sneaky generator and critic model, we make slight modifications to the prompts mentioned above avoid including incorrect steps that should be in the LLM output and unnecessary information, such as draft critiques. As shown in Figs. 7 and g we demonstrate the prompt templates for training the sneaky generator and critic model, respectively. These templates also remain unchanged during testing, data generation for self play, and reinforcement learning processes.
C More Details
C.1 Evaluation Details
PRM800K 23 is dataset collected by OpenAl for training and evaluating process supervision models. It is large in scale, containing 800K GPT-generated reasoning steps with numan-annotan correctness. Additionally, PRM800K includes many pairs of correct and incorrect steps that share common partial solution. Therefore, we construct 1,341 pairs of steps from the test split to evaluate model performance.
step correctness, but it only includes test set for evaluating models. Compared to PRM800K, the reasoning steps in BrocessBench are more diverse, comprising 3,400 cases from 12 different LLMs. All of these are math problems sourced from four datasets, including GSM8K [56] MATH [54] OlympiadBench [57] and Omni-MATH (58). For incorrect solutions, we only retain the first incorrect step. while we randomly sample one correct step from correct solution. We then feed the mixed 1, 1,700 correct steps and 1,700 incorrect steps along with their corresponding partial solutions into the critic models.
ProcessBench (27) is a benchmark with
DeltaBench 35 is the newest process benchmark focusing evaluating long CoT collected from different open-source reasoning models, such as RI 21 and QwQ [22]. We only utilize the math-related problems in this benchmark evaluate model performance. Similarly, retain the
21

---

labeled erroneous steps and sample the same number of correct steps, totaling 1,542. Given that existing PRMs baselines and our adversarial data generation process only collect short CoT data, this benchmark is more challenging and can be utilized to evaluate the effectiveness of our critic models in generalizing to popular reasoning models with long CoTs
MATH500 [23] 54] and AIME2024 [37 are two highly popular benchmarks used to assess the mathematical reasoning abilities of LLMs. The former consists of 500 competition-leve math problems, while the latter is derived from the American Invitational Mathematics Examination 2024. We evaluate the performance of LLMs on these two benchmarks when assisted by different verifiers in reasoning.
C.2 Preparing Data
For the SFT phase of the critic, we utilize the reasoning process data from PRM800K, along with prompting GPT-4 and DeepSeek-RI-Disill-ONee
[21]. to generate the required step-level critique data. We employ human annotations from PRM800 to filter out correctly generated data, ultimately obtaining 21.8K data, including 9,4K correct steps and 12.4K incorrect steps. As for the sneaky generator, we also prompt GPT-4 to teach the LLM to transform the correct steps from PRM800K into incorrect steps, finally collecting 13K data for SFT.
During the self- play phase, we use problems from the training set of PRM800K 23 to generate adversarial data for reinforcement learning. We use a total of three types of LLM solvers (Llama- 3. Instruett [10] Qwen2.5-7B-Inme Instruuttt and Qwen2.5-322 --n.2.5-33B-nntruuue [12 to provide the initial reasoning steps, in order to sample the correct steps and perform a sneaky transformation, which are then fed to the critic for adversarial self-play. Since we need the LLM to complete from both the original correct step and the sneaky step generated by the sneaky generator and compare the problem-solving success rate to determine whether the sneaky step contains an error that can truly affect the reasoning process, we first pre-generate 4 solutions for each problem and filter out those that have a success rate of O, which are inherently unsolvable. For the remaining problems, we consider those with success rate of 1/4 and 2/4 medium difficulty level for this LLM, while those with a success rate of 3/4 and 4/4 are considered relatively casy. We primarily use medium-level problems to construct the training data, which ultimately accounts for 90% of the dataset, while easy problems are retained at about 10% because the model can already solve these problems smoothly without much additional learning and we only need a small amount of such data. These filtered medium-level problems will have 16 solutions generated by each LLM solver and easy problems will directly use the pregenerated 4 solutions. For each correctly predicted solution, one correct step is sampled and performed sneaky transformation. The successfully transformed incorrect steps are then further filtered for adversarial self-play.
The first round of self- occurs between two SFT models (sneaky-O and critic-O). We collect 6.4K data for the critic model for reinforcement learning, with 1:1 ratio of positive to negative samples. Meanwhile, the sncaky generator receives 6K data, divided equally (2K each) into three scenarios: failing to attack the LLM solver, successfully attacking the LLM solver but losing the critic, and successfully attacking the LLM solver while defeating the critic. The collected data in round 1 help us iteratively update the models to sneaky-1 and critic-1. As mentioned in Sec. [3.5] we balance the adversarial game to collect training data. Therefore, the second round of self-play occurs between sneaky-1 and critic-O. We further collect 6.8K data for the critic model, maintaining 1:1 ratio between positive and negative samples, while continuing to gather 6K data for the sneaky generator, with the three scenarios still evenly distributed at 1/3 each. Finally, the data from two self-play rounds is merged to conduct offline reinforcement learning on sneaky-O and critic-O, updating them to sneaky-2 and critic-2, respectively.
C.3 Training Hyperparameters
In the SFT initialization phase for both sneaky generator and critic models, we employ a batch size of 64 and a learning rate of 5e-6. We train the models for 3 epochs, with the maximum sequence length set to 4,096. To ensure both stability and convergence during training, we also incorporate KL penalty into the training loss, setting the KI. coefficient at O. During the reinforcement learning of the self-play phase, we keep the batch size as 64 but use learning rate of 26-6. Except for setting
22

---

the KL coefficient at 0.1, we also add an SFT loss with a coefficient of 0.15 to ensure the stability of RL training.
D Case Analysis
present comparison of critiques provided by two SPCs trained in round and round 2. The input to the SPCs includes not only the system prompt (Sec. B but also the problem, partial solution, and the last step of the current reasoning process within the blue box in the figure. The last step contains logical error, mistakenly identifying two expressions as unmatched. The critique from the round 0 SPC considers the last step be correct, agreeing with the view that the expressions are inconsistent. However, the round 2 SPC, evolved through self-play training, accurately identifies the type of error, namely logical error (underlined in the figure), recognizing that the two sides of the equation can be equivalently substituted. The entire analysis process is clear and coherent, ultimately leading to the correct prediction that this step incorrect.
As shown in Fig.
23

---

User Prompt
You math teacher. Given
partial solution, you the step of
incorret slep,
while ersuring the incorrect subtle enough
everlooked.
You should follow these
corver the last step:
1. Analyze the given partial solition and the last step. Clearly explain the salving process in the last stop.
2. Choose appropriate error type from the "Predefned Eroo Types" complete the emror generation. Specify your error neration method
based on the currant case, making the orror loss notccable..
3. Sep-by-step. write out the detailed emor generation process for converting the Correct Last Step into Incorrest Step.
4. Wrap the final incorret step with <Answer> </Answer:
question, you be given
step.
stap, ""you must
you
step during process!"
# Predefined Emor Types
Error Type Logical Error
Reference cases:
incorrect orkentation of geometric figures
systematic counting enor
incomplete and inaccurate fellig of factors
Incorract intorpratation and connection of definitions
Error Type calcuistion error
erfeeence cases:
make incomrec simpification
Incomect
invalid algebraic operation
Error Type Misunderstanding the Conditions.
Reference cases:
Use imcomplete Condition
Use contradictory condition
misinterprecaation of problem requirements
Error Type Use Incomect Malas/FomuuisPrpptttt
Reference cases:
incorrect identiiicaion of prime numbers
misapplication propertics complex numbers
introduction
inequality
Type Approach
Refe
incompl solution
Your Task
## Problom (problem)
Partial Salution solltion))
Correct Stap (correct step)
(Raleeancc)) Incorrect Next (incorrect
## Your response
Figure 4: Prompt for querying GPT-4 to collect raw data of sneaky transformation CoT.
24

---

User Prompt
You are heloful oriic. Given math Problem, Partial Solutin, the current Laet Step of the solution, You need to provide criiquue for the coroctnees the Last Step.
You need response slep-by-slop analysis:
Analyzing the general thought
Orlique,
critique
logical coherence summary general thought
Partial Solution.
3. Conclusion. the and the respanse, (ooree)((orrect) liboxed (bxoae((mmaarrct))
corretnnes the Last Step.
Problom
(arablemi
Partial Solution (pertial solution)
Step (last_etep) step)
Figure 5: Prompt for querying DeepSeek-RI-Distill-ONee to collect raw critiques with long CoT.
System Prompt
You are helpful ontic. Given math Problem, Partial Solution, the current Last Stop of the solution, You need to provide criiquu for the correciness of the Last Step. You already have Draft Criliuee, so you only need rewrite it clesrer and more concise format.
You need to response a step by-step analysis:
Analyzing the general thought the Partil Solution. If the Draft Critique includes this enelysis, you can directly summarize from it.
Crtique.
version ortique here. cument Crtique
may contain
extensive consideration attempts derivation analysis
You only
analysis
Step
This should maintain caherence with the summary the general thought Partial Solution.
Condusion. Please draw condusion about the corectrees the Last Slep. Based enalysis the provided Draft rilique, determine whethar the Last Step is cornec or incomrect it incorrect, you should also summarize spectfcc type of error, such as calculation enror, logical error etc. At the end of the response, cutput -lnsserrCoomettt///ree or Awwswer-incoreee///Aaaaaa to represent the correctness of the Last Slep.
NOTE:
crtique, pretend you
beforehand, avoiding phrases crttiuee.
Problem, Partial Solution,
Step.
your revised of you only need focus Lost
nacassary
obtaii final
DO NOT conclusion first and then the explanations for Instead, Criqque, you start with analysis of Last Step. Then, Conclusion, drawing conduusio about whether the Last Slep correct incorrect.
4. The current partal solutio may be incompiate, which not an ertor type. You only noad focus on the correcnnss of the Last Stop.
User Prompt
(problem)
#e Partial Solution (pertial_sok solution
#a Last Step (last step)
Cittique
(draft, ontique)
Figure 6: Prompt for querying GPT-4o to rewrite long critique into a brief and standardized critique.
25

---

System Prompt
problem partial solution, you need convort stop partial stop, enough easlly
ensuring
You should follow these
the last
Aralyze the given pertial solution end the last Clearly explain the solving the
Choose appropriato error type from the "Predefined Error Types" complete the error generalion. Spacity arror gemeeration method besed on the current case, making the eiror less notioeable.
Step by-stop, write out the detailed error gonoration process for convorting the Cormact Last Step into the Incorrect Step. Wrap the incomrect step with <Answers </Arswer>:
Typos
Type Logical
Reference
orientation geometric figures
syelemafic counting emror
incomplees cio and inacourato listing of factors
Incorrect interpretation and connection of definitons
Error Type calculation error
Referonce cases:
make incorrect simplifcation
algebraic operation
Error Type Misunderstanding Conditions.
Reference
Use Imcomplete Condition
Use contradictory condition
misinterpreiasion of problem requirements
Error Type Use Incorrect RulesFomuless Pomulesssrroorrr Reference cases:
incorrect dffcatin of prime numbers
missppiication properties complex
introduction irreevvant inequality
Error Incorrect Approach
Reference cases:
incompiett ent salution
User Prompt
Your Task
(problem)
Ail Partial Salution (partiel_ solution)
#W Correct Last Step (porrect_lest _esp__eepp)
Your response
Figure 7: Prompt training the sneaky generator.
26

---

System Prompt
You helpful ontic. Given math Problem, Partal Solution, sorreciness the Last Step.
Step of solution, You need provide crique for
You need to response step-by-stes step arelysis:
1. Analyzing the general thought of the Partial Solution.
Critque. You should write brief crtique hore. This part should also maintain logical coherence with the summary of general thought the Partill Solution.
Conciusion.
response, output Correct-/Answerr> or <hnswee-inccreekkkksseee represent correctness
Slep.
User Prompt
HR Problem
(problem)
Partial Solution
Last Step (lest, step)
Figure 8: Prompt for training the critic model.
SPC Input
Atf Problem
Find postive integers $n, k_1, Idols, k_nS such that Sk_1 lodole: K_n 5n-45 and WraottHH_1) 1)+1 lodots + Vrsa[]) [)) - 1.
Partial Solution
salve the problem, we need to find all positive integers V[ n, k. Idots, () such that: First, K lodats k. fin 4 and Second, (frro/1kK_1 13 Vrao[1]K 2) todots Mraao = We start by analyzing second condition. The equation Mirroo Mrrof1HK
Vrao[1](K well-known example Egyptian fraction representation.
suggests
2. 1) denominators
sum equals
Last Stop
For small values of However, Vrr 1) does not satisfy
find solutonn manuelly. First, consider the case V( I then -41)-4--110
SPC Critique (Round 0)
SPC Critique (Round 2)
AA# Analysis the General Thought of
8BH Analysis of the General Thought of the Partial Solution
The partial solution begins by analyzing the given condition: 1( 1 k_2 k_ () and (rra/1] 1) (rac(1)((k Vraof1XK_n W.
The partial solution begins by analyzing the given condition:: 1( K_1
k Ifrac/1kk__ n) suggests distinct unit fractions. The soi colution then proceeds test small values of V find potantial salutions.
V. Vimo[1)kk 1) (racff)(k 2) the
condition
correctt
Egyptian fractions that add 1. The salution proceeds to test small values V. () to find potential solutions.
suggests
sume
Egypöan
#d# Crttique
#až Cittique
The last step the partial solution corsiders the case V = 1 calculates V. k_1 5(1)-4 4= 1) and then checks V Vrao[1Kk 1)
In the lest step. the solution considers the case = correctly calculates V k_ 5(1) 4 11). However, it then incorrectly
salisfiee the condition 1_1 1-11 1.. The solution correctly identifies that this does not sadisty the equation since V k_1 must be 1, but the sum the reciprocals would 1, which is consistent but does not meet the requirement of distinct denominaatos V nm
conciudes that Vrac[1XX 1) - does not satisfy k_1 1. This is logicall error. The equation Mrrroffjk, 1) 1 does indeed satialy X K_1 1. substitutin V k_ 1 into the equation gives V, Vrac(1)(1) which is true.
#A#
eMN Conclusion
The comectness of the last stop <Answer> CowneettoorrrrttAAAAAa.a
The correctness of last stop <AnswerrinnommeekAAnneee
Figure 9: SPC critiques on ProcessBench before and after self-play training.
27


Reviews:
[{"review_id": "kVbcJ2I9Ru", "reviewer": "Reviewer_d2Hw", "summary": "The key idea of this paper is intuitive: leveraging a self-play critic to improve the quality of the LLM reasoning process. The experiments show substantial improvement on several benchmarks, even after just a couple of iterations. The paper is well written and easy to follow.", "questions": "The major question is that do you have more insights on the rounds of self-play. What's would happen if it continously increase?", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The idea of this paper is intuitive and it should be a direction to explore  to further improve the LLM reasoning. Though the submission is not in your area, I expect that there should be some related works that not well discussed in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your comments that our idea is intuitive, our experiments show substantial improvement, and the paper is well written and easy to follow. We have addressed your question below.\n\n**More insights on the rounds of self-play**\n\nThank you for your question. We believe that the essence of adversarial training in our SPC lies in fully exploring and mining the potential erroneous reasoning steps that may arise when attempting to solve certain problems, and using this data to train the sneaky generator and critic model. Considering the balance between data collection efficiency and performance gains, conducting 2-3 rounds of iteration is appropriate to fully utilize the problems in the MATH dataset. In addition, we think that the rounds of iterations may also be related to the difficulty of the problems. For more challenging mathematical problems or other complex tasks (e.g., training general agents), it may be worthwhile to conduct more rounds of exploration, as we could continuously generate more diverse data from difficult problems. We will provide more insights and analysis in our revised version."}, {"review_id": "vCLBFRnxeB", "reviewer": "Reviewer_czJQ", "summary": "This paper proposes Self-Play Critic (SPC), a novel framework for training a step-level verifier for CoT reasoning. SPC uses an adversarial self-play game between two roles: a sneaky generator that injects subtle reasoning errors, and a critic that identifies them. Through reinforcement learning on the outcomes of these interactions, both models co-evolve. Experiments across reasoning benchmarks (ProcessBench, PRM800K, DeltaBench) demonstrate the superior performance of SPC.", "questions": "The evaluation process on PRM800K and DeltaBench seems to deviate from their original settings (as noted in lines 244–246).  Could the authors clarify the specific modifications made to the evaluation process and justify how these changes may impact the comparability of previous PRM methods?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The self-play setup at the step-level is original and well-motivated, filling a clear gap in current PRMs.\n\n- The paper includes solid and comprehensive experiments across several benchmarks. SPC shows consistent improvements across all.\n\n- The paper is well written.\n\n\nWeakness:\nIn my view, the paper does not have any major fundamental issues. The only issue is that ORM baselines should also be considered in BoN-style evaluation, such as [1]\n\n[1] Generative Verifiers: Reward Modeling as Next-Token Prediction", "was_revised": true, "final_justification": "No more rebuttal is made by authors. I maintain my original assessment.", "rebuttal": "Thank you for considering our approach to be original and well-motivated, with solid and comprehensive experiments, and for your positive comments on the writing of the paper. We have addressed your questions below.\n\n**W1. ORM baseline**\n\nThank you for your suggestion. We have checked several representative generative ORMs [1, 2, 3], including the Generative Verifiers work you mentioned. Unfortunately, none of them have released their trained model weights. Therefore, we found another open-source ORM on Hugging Face, which was trained using Deepseek's data. We compare the performance of this new ORM baseline with other baselines as well as our proposed SPC. For example, on Qwen2.5-32B-Instruct solver, the test results are as follows:\n\n| Methods    | MATH500 | AIME2024  |\n|------------|:---------:|:-------------------:|\n| ORM     | 78.2   |   13.3    |\n| Math-Shepherd     | 78.8       | 13.3  |\n|Qwen2.5-Math-7B-PRM800K | 82.8 | 16.7 |\n| Self-Consistency + ORM | 82.4 | 16.7 |\n| Self-Consistency + Math-Shepherd | 80.8 |13.3 | \n|Self-Consistency + Qwen2.5-Math-7B-PRM800K| 84.6 | 16.7 |\n| SPC     | 83.0   | 16.7 |\n|Self-Consistency + SPC | 85.2 | 23.3 |\n\n[1] Zhang et al. \"Generative Verifiers: Reward Modeling as Next-Token Prediction.\" ICLR 2025.\n\n[2] Ye et al. \"Beyond Scalar Reward Model: Learning Generative Judge from Preference Data.\" arXiv 2024.\n\n[3] Mahan et al. \"Generative Reward Models.\" arXiv 2024.\n\n**Q1. Evaluation settings**\n\nTo better reflect the practical effectiveness of PRMs/Critic models in test-time search, we adopt a different evaluation setting. This is because, during test-time search, it is more important for the critic model to analyze the current step and promptly correct potential errors in this step. Thus, only the current step and the preceding reasoning steps (partial solution) will be used as input, rather than analyzing each step in the entire solution from the beginning. In this setting, PRMs are inevitably affected due to their inherent limitations of typically providing step-level scores on the complete solutions. Therefore, on both PRM800K and DeltaBench, we further compare against a wide range of baselines that prompt powerful LLMs as critic models. The results demonstrate that our SPC significantly outperforms these baselines."}, {"review_id": "c0k3KbHL1m", "reviewer": "Reviewer_JDJJ", "summary": "This paper proposes Self-Play Critic (SPC), an adversarial training framework designed to learn a more robust and effective process-based critic model. The core of SPC involves augmenting a standard critic model with a \"Sneaky Generator.\" This generator is tasked with a specific adversarial goal: given a partially correct solution, it is trained to produce a plausible but incorrect subsequent step, aiming to deceive the critic. The critic, in turn, is trained to maximize its ability to discriminate between correct reasoning steps and the subtle errors introduced by the generator. Both the critic and the generator are optimized using reinforcement learning in a self-play loop. The authors validate the effectiveness of the resulting critic model on standard reward benchmarks and demonstrate improved performance with test-time scaling.", "questions": "Please refer to the weaknesses part.", "limitations": "Please refer to the weaknesses part.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The proposed adversarial self-play approach is a novel contribution to the field of process-based feedback. By generating its own challenging training data, the method effectively improves the critic's ability to detect nuanced errors, moving beyond the limitations of static datasets.\n\n2. The SPC framework helps to alleviate the well-known bottleneck of acquiring large-scale, high-quality human preference data for training process reward models. By synthetically generating difficult negative examples, it amplifies the utility of an initial seed dataset.\n\n3. Clarity and Structure: The paper is well-written and logically structured, making the complex interplay between the generator and critic accessible and easy to follow.\n\n## Weaknesses\n\n1. A notable limitation is that the SPC framework still requires a pre-existing, human-annotated dataset (e.g., PRM800K) to bootstrap the training process. This dependency means that SPC does not entirely solve the foundational problem of data acquisition for PRMs but rather acts as a data amplifier and refiner. The quality and domain of the final critic are likely still heavily influenced by the initial seed data.\n\n2. The experiments appear to be conducted with a specific choice of model architecture (Qwen) for the critic and generator. The paper would be more persuasive if it included ablation studies exploring the framework's performance with different underlying models (e.g., using Llama 3.1 for both components). This would help ascertain whether the benefits of SPC are generalizable across various model families and sizes or if they are dependent on a particular architectural setup.\n\n3. SPC involves iterative adversarial training and reinforcement learning, which appears to be computationally intensive. The paper does not provide details on the training time, computational resources required, or the overall efficiency of the self-play loop. This information is crucial for assessing the practical viability and scalability of the SPC framework. It is recommended that the authors clarify these costs in the main text.\n\n4. The results from the test-time scaling experiments raise potential concerns about the generalization capabilities of the learned critic. The performance improvement on the AIME dataset seems considerably smaller than on GSM8K. A plausible hypothesis is that this discrepancy arises because the bootstrap dataset (PRM800K) is collected from solutions to GSM8K problems.", "was_revised": true, "final_justification": "The authors' rebuttal clearly addressed all my concerns especially for SPC's generalization abilitity to other models. Thus, I have raised my score from 4 to 5.", "rebuttal": "We are very pleased to hear that you consider SPC a novel contribution that alleviates the well-known bottleneck of data, and we appreciate your recognition of our efforts in making the complex interplay between the generator and critic accessible and easy to follow. We also address your concerns below.\n\n**W1. Human-annotated data**\n\nWe did utilize a portion of human-annotated data (PRM800K) to perform a cold start for the model, which is a common training stage prior to RL training. It is reasonable to use these seed data, because the 7B model we use does not have sufficient initial capabilities as a sneaky generator or critic. If a more capable model were used, our framework could theoretically skip this cold start step and proceed directly to RL training. Directly enabling a stronger model to self-evolve is indeed a valuable direction to explore.\n\nBesides, in RL training phase, we only utilize our self-play framework to generate new data and do not rely on any human data. It is also worth noting that the PRM800K dataset consists of human annotations based on early GPT-4 outputs, and the resulting PRMs are thus tailored to the style of GPT-4 outputs. By continuously generating data through self-play, we successfully generalized our approach to multiple types of solvers, including DeepSeek-R1-Distill-Qwen-7B, Qwen2.5-32B-Instruct, and Llama-3.1-8B-Instruct. Our method makes full use of existing publicly available data and mitigates the influence of the initial data to some extent.\n\n**W2. Training different models**\n\nWe chose the current base model, Qwen2.5-7B-instruct, because it has relatively strong capabilities in this model size. Some contemporaneous works [1, 2] have also made similar choices. In addition, we have also verified that our SPC can effectively improve the performance of different solver types (Llama3.1, Qwen2.5, and Deepseek-R1), which to some extent demonstrates the generalization ability of our framework. Thank you for your comments, and we will also try to verify the generalizability of our SPC by training other model families, such as Llama.\n\n[1] Tang et al. \"RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback.\" arXiv 2507.\n\n[2] Yang et al. \"DeepCritic: Deliberate Critique with Large Language Models.\" arXiv 2505.\n\n**W3. Details of Experimental Costs**\n\nDuring the data collection phase, we used 8 H20 GPUs to collect offline data, and each round of self-play took approximately 8 hours. For training, our final critic model used about 13.2K data samples for offline RL and required 3 epochs of training. When we used a cluster with 32 GPUs, training a single model takes about 2.5 hours, whereas using 8 GPUs takes about 8 hours.  Besides, our framework only needs to use offline RL, which requires less computational resources and time compared to recent work using online RL. Overall, the resources required by our SPC framework are moderate.\n\n**W4. The generalization capabilities of the critic model**\n\nThank you for your thoughtful consideration. First, we would like to kindly point out a small typo in your comments: our experiments were conducted on MATH500, not GSM8K. We have also noticed the difference in performance gains of SPC on MATH500 and AIME2024. However, we believe that this does not necessarily indicate weaker generalization on AIME. In fact, the experimental results on Qwen2.5-32B-Instruct show the opposite. The result of Self-Consistency + SPC on MATH500 is 85.2%, which outperforms the 80.8% achieved by Self-Consistency + Math-Shepherd. On AIME2024, their performances are 23.3% and 13.3%, respectively. Obviously, when assisting Qwen2.5-32B-Instruct in problem solving, SPC brings a more significant performance improvement on AIME2024.\n\nWe also agree with your hypothesis. In our experiments, the training dataset we are using is not very large in scale or high in difficulty. Recent work [3,4] has shown that increasing the difficulty and scale of the data is beneficial for further improving model performance. We believe that training our framework on larger-scale and more challenging data will lead to better results on more challenging test sets such as AIME.\n\n[3] He et al. \"DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning.\" arXiv 2025.\n\n[4] Hu, Jingcheng, et al. \"Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.\" arXiv 2025."}, {"review_id": "F8wJXToZec", "reviewer": "Reviewer_pEmR", "summary": "This paper tackles the problem of judging every single step in an LLM’s chain-of-thought without human labors. The authors propose a self-play framework—aptly named Self-Play Critic (SPC)—in which one model learns to generate sneaky & believable mistakes, while its adversarial model learns to catch them. \n\nThery start with PRM800K’s human-labelled step pairs to give both “sneaky generator” and “critic” a supervised warm-up. Then the generator learns to rewrite correct steps into subtly wrong ones (five error archetypes) based on the previous steps, and only those sneaky steps that make a downstream solver fail hard (success rate drop from ≥ 75 % to 0 %) are kept. Both gold and sneaky candidate steps are fed to the critic, which obtains reward if it spots the flaw, and otherwise the sneaky generator scores. Each win/loss is a ±1 reward. They store every interaction and run offline RL with KL anchoring so neither model drifts too far from its SFT roots. \n\nAfter two self-play rounds the critic’s step-level accuracy climbs from 70.8 % to 77.7 % on ProcessBench and posts similar gains on PRM800K and DeltaBench, beating existing process reward models. When plugged into test-time reasoning, SPC nudges three different solvers to new highs on MATH500 and AIME 2024—e.g. a 7 B distilled-R1 model hits 94 % on MATH500 when guided by SPC versus 91.8 % with the best baseline verifier. \n\nOverall, the work shows that step-level feedback can bootstrap itself: once you have a seed set, two small models can keep generating better supervision for each other, all without further human labeling.", "questions": "1. Why insist on the “≥ 75 % vs 0 %” success-rate gap? Did other cut-offs (e.g. 60 %→10 %) hurt critic learning or just add noise?\n\n2.  All filters and rewards hinge on the three chosen solvers. How does SPC behave if the solver is weaker/stronger or writes in a very different style?", "limitations": "Yes.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 1, "strengths_and_weaknesses": "S1. Balanced, asymmetric evolution strategy: Authors recognise and solve the “generator too weak” problem by pairing the new generator with the previous critic; keeps win-rate ~30 % and drives continual improvement. Provided experimental support for the importance of balancing, which is not covered in similar adversarial training papers.\n\nS2. Fine-grained error taxonomy: Five concrete error families (logical, calculation, mis-reading conditions, mis-using rules, wrong approach) give the generator a structured search space and make the critic’s feedback interpretable.\n\n--- \n\nW1. Experiments confined to math; it is not clear whether similar sneaky generation strategy would be effective for more ecologically vaild & environment-dependent tasks, such as software development, web agent, os operation. \n\nW2. Reward shaping still heuristic (binary ±1); sensitivity analysis limited.\n\nW3. The current generator-verifier mutual enhancement effect from adverarial training methodology has been long known for reasoning tasks even before the age of LLM, which significantly undermines the originality of the work. The most important takeaway from this paper is perhaps the sneaky data synthesis strategy -- but the error taxonomy limits the domain transferability.", "was_revised": false, "final_justification": "", "rebuttal": "We are pleased that you praised our introduction of the evolution strategy, which is not covered in similar adversarial training papers, as well as our implementation of a fine-grained error taxonomy. Below, we have carefully addressed your questions.\n\n**W1. Evaluation Tasks**\n\nThank you for your comment. We also discussed this issue in the Limitations section (see Appendix A). We acknowledge that our current experiments are limited to mathematical tasks, as this domain provides high-quality, step-level human annotations that allow us to effectively evaluate the performance of our trained critic. **We also believe that this framework could be highly valuable in other domains**, such as the agent scenarios you mentioned. For example, the outcomes of multi-agent interactions can also be used to generate data for reinforcement learning. A well-trained critic can effectively assess the agent’s actions at each step and guide the decision-making process. Constructing high-quality evaluation benchmarks and extending our methods in these domains can serve as our future work.\n\n**W2. Reward Shaping**\n\nOur framework focuses on the design of the RL task and environment, and the reward signal is solely determined by whether the LLM wins or loses the game. Therefore, we adopted a binary reward (+1 for the winner, -1 for the loser) to directly reflect the competitive nature of the game and to ensure a clear learning signal for both models. **This reward design was also used in the self-play of AlphaGo Zero [1], demonstrating its robustness.** Thank you for your comments, and we will provide additional explanations and clarifications in our revised version.\n\n[1] Silver, D., Schrittwieser, J., Simonyan, K., et al. \"Mastering the game of Go without human knowledge.\" Nature 2017.\n\n**W3. The originality of SPC**\n\nTo the best of our knowledge, related research in self-play or adversarial training methodology can be categorized as follows:\n1. Training in board games. For example, AlphaGo Zero [1] continuously collects data through self-play in the natural Go environment with the help of MCTS, using both the action probability distribution and the overall win rate estimation as learning targets, and updates the network via reinforcement learning.\n2. Self-play learning for LLMs through existing games to achieve capability transfer [2, 3]. For instance, SPAG [2] leverages a party game called Taboo, where the LLM is required to guess a target word based on dialogue information, thereby improving the LLM’s reasoning ability.\n3. Test-time self-play among LLMs without any training [4, 5]. For example, rStar [4] predefines a set of actions and combines them with MCTS for reasoning trajectory search, thus achieving a mutual generation-discrimination process and improving mathematical reasoning performance. This is indeed a solver-critic self-play during the testing phase to improve performance, but it cannot enable LLMs to self-evolve through self-play.\n4. Some early GANs [6, 7] achieve great success in image generation through adversarial training, but it is difficult to transfer this approach to NLP tasks. This is because images are continuous signals, which makes it easier for GANs to optimize the generator, allowing the generated images to gradually approach real images by minimizing the cross-entropy loss on the discriminator’s real/fake logits. However, for discrete tokens in text tasks, the optimization methods used by GANs are difficult to be effective.\n\nIn contrast, the originality and contributions of SPC lie in:\n1. Compared to AlphaGo Zero, SPC extends the self-play approach to the recently popular domain of LLM reasoning. The main innovation is the careful design of **an adversarial training process for LLM tasks that lack a native game environment**, enabling continuous data generation for reinforcement learning.\n2. Compared to SPAG, in the design of adversarial games, we do not simply utilize existing board or party games to indirectly improve LLM capabilities. Instead, SPC **directly designs a step-level generation and verification game for the critic**, enabling direct training of the LLM’s critic ability.\n3. Compared to rStar, we combine the self-play framework with reinforcement learning to improve the model’s capabilities, rather than prompting LLMs and only performing test-time computation. Our newly developed self-play training framework **enables the critic to self-evolve through self-play**.\n4. Our SPC is also fundamentally different from the adversarial training framework of GANs on vision tasks. For text tasks with discrete tokens, we **leverage reinforcement learning to achieve adversarial learning**. Here, the sneaky generator’s goal in RL is not only to attack the critic, but also to generate more subtle errors. We finally obtain a step-level critic for assessing LLM reasoning and providing valuable feedback. This is fundamentally different from the mutual improvement framework of adversarial training in GANs.\n\nThank you for your constructive comments. Clarifying the difference of SPC to previous related adversarial training methodology helps better highlight our contribution and improve the quality of the paper. We will add these discussion into the Related Work in the revised version.\n\n[1] Silver et al. \"Mastering the game of Go without human knowledge.\" Nature 2017.\n\n[2] Cheng et al. \"Self-playing Adversarial Language Game Enhances LLM Reasoning.\" NeurIPS 2024.\n\n[3] Wu et al. \"Enhance Reasoning for Large Language Models in the Game Werewolf.\" arXiv 2024.\n\n[4] Qi et al. \"Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers.\" ICLR 2025.\n\n[5] Xi et al. \"Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision.\" arXiv 2024.\n\n[6] Goodfellow et al. \"Generative Adversarial Nets.\" NeurIPS 2014.\n\n[7] Karras et al. \"A Style-Based Generator Architecture for Generative Adversarial Networks.\" CVPR 2019.\n\n**Q1. Success-rate gap**\n\nYes, your guess is correct. Setting a relatively strict threshold is intended to improve data quality and reduce noise. During preliminary experiments, we explored other success-rate thresholds (e.g., 50% and 25%), but observed that these settings resulted in less stable training. We suspect this was due to the introduction of additional noise or potential reward hacking. Our SPC framework allows us to efficiently generate large amounts of raw data and obtain high-quality training data through filtering. Therefore, raising the threshold is relatively beneficial and harmless.\n\n**Q2. Solver selection**\n\nIn fact, we have already considered both the performance and output style of the solvers in our selection process. \n1. Our SPC can cover both weak and strong solvers  (see Table 3). Specifically, on AIME 2024, Llama-3.1-8B-Instruct achieves a maximum performance of 6.67%, while DeepSeek-R1-Distill-Qwen-7B reaches 73.3%, obviously demonstrating their performance gap. \n2. Our SPC can generalize to different output styles (see Lines 283-285). We only used Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, and Qwen2.5-32B-Instruct to collect the original solutions, which were then fed to the sneaky generator and critic for adversarial training. For testing, however, we applied SPC to DeepSeek-R1-Distill-Qwen-7B, a solver with strong reasoning capabilities and a completely different long CoT output style. On MATH500, compared to the PRM baseline result of 91.8%, using SPC to assist the R1-style solver achieved a performance of 94.0%, demonstrating an obvious improvement."}, {"review_id": "CRfFDY00iq", "reviewer": "Reviewer_oHJP", "summary": "This paper proposes a method for training a step-wise verifier/critic model for math reasoning CoTs. The method, Self-Play Critic (SPC), is trained via an adversarial 2-player game, where a generator is trained to introduce errors into reasoning while a critic is optimized to find the errors. Both the generator and critic are initialized with SFT policies using data created from PRM800k, and then trained via RL, with the critic receiving a reward for detecting a mistake and the generator receiving a reward for fooling the critic. \n\nThe method is used to train a Qwen-2.5-7B model, which is evaluated intrinsically on three PRM benchmarks (ProcessBench, PRM800k, and DeltaBench) where it shows improvements over both trained PRMs and advanced LLMs prompted to act as critic models. The method is evaluated extrinsically on 2 math reasoning tasks (MATH500 and AIME) where it is used in an overgenerate-and-filter approach (optionally combined with self-consistency). On the intrinsic evaluation, the method shows robust improvements over existing methods, while the size of the gains on the downstream evaluation is somewhat smaller. The ablations test matching generators with critics from earlier rounds, as matching from the same round makes the task too easy for the critic.", "questions": "1. Does SPC training generalize to other models besides Qwen?\n2. Does SPC-trained Qwen benefit other models downstream?\n3. What explains the relatively small performance improvements on the dowstream eval?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- **Intuitive and useful idea**: The framing of step-wise verification as an adversarial task makes intuitive sense, and showing how these two models can be optimized via RL successfully is a useful contribution.\n- **Outperforms PRMs**: on intrinsic evals of PRMs, the model trained via SPC outperforms several baselines, including proprietary models being prompted as critics and specialized PRMs. \n- **Sensible analysis and useful details**: I appreciate that the paper shows improvements across rounds and has a clear analysis of what does (and does not) work in terms of pairing the sneaky generator with the critic at the same round vs. the prior round's critic.\n- **Clear writing/method description**: The paper is easy to follow and well-written.  \n\n**Weaknesses**:\n- **Generalization ability is unclear**: There are a couple questions I have about generalization which are left unanswered:\n1. Does the method generalize to other models? The paper only trains one model (Qwen-2.5-7B-Instruct) and so it's not clear to me this would generalize to other models. It would be helpful to see results on other similar models to know that the method is not overfitting to one model.\n2. Can a critic of one model type help other models at test time? As far as I can tell, the results in Table 3 are using Qwen both as the critic (trained with SPC) and as the reasoning model. While I understand that data was mixed from different models for training, it would be nice to see transfer here to other models. This wouldn't require any training, since you could use the SPC-trained Qwen model to do reranking on Llama or another model. \n- **Extrinsic results are underwhelming**: The size of the gains in Table 3 are quite small, and in some cases using SPC without SC has lower performance than PRM800k trained Qwen (e.g. on AIME2024 with DS-R1-Distill-Qwen, though that may be due to small sample size in AIME). Further explanation of why the gains here are not as substantial would be helpful. \n- **Further uses of SPC could be explored**: The main downstream use explored here is for a resampling approach, but PRMs can be used in more advanced ways, e.g. for refinement (https://arxiv.org/abs/2409.12147). \n\n**Other comments**\nGiven the adversarial training nature of the method, I would have liked to see some more discussion on GANs", "was_revised": true, "final_justification": "After reading the author response, I am raising my score, as most of the points raised above have been addressed. While I see the some concerns raised by other reviewers (limited to math, requiring startup data) as valid limitations, I think that these are shared by many other papers and this paper represents a step towards an adversarial data generation pipeline (while maybe falling short of showing that the solution works on a comprehensive set of diverse tasks).", "rebuttal": "We appreciate that you found our ideas intuitive and useful, achieved better results than PRMs, provided sensible analysis and useful details, and offered clear writing and method descriptions. We have also addressed your questions below.\n\n**W1. Generalization ability**\n\nFor the critic model, we chose to train on Qwen2.5-7B-Instruct because it is one of the most popular and representative models, which allows us to quickly verify the effectiveness of our proposed SPC. In addition, many of our baselines are also 7B models, so using 7B models for training makes the comparison fairer. We will try to include results on additional models to further demonstrate the generality of our approach. \n\nYou also suggest performing experiments where the critic model is used to assist other types of models during testing. We would like to clarify that this is **exactly what we did in the experiments shown in Table 3**. In the leftmost column of the table, we are referring to using the SPC model to assist different solvers in problem-solving, rather than using these different solvers to collect data. Specifically, we used three solvers for experiments: Llama-3.1-8B-Instruct, Qwen2.5-32B-Instruct, and DeepSeek-R1-Distill-Qwen-7B. These results demonstrate the generalization ability of the proposed SPC, as it can help different types of solvers further improve their performance. \n\n**W2. The performance of SPC in assisting different LLMs**\n\nThe effect of our SPC in assisting DeepSeek-R1-Distill-Qwen-7B on the AIME2024 dataset is indeed not very significant. As you mentioned, this may be due to the small sample size in AIME, which introduces some randomness. Another possible reason is that AIME2024 is quite challenging, and a performance of 73.3% is already very high, making further substantial improvements difficult. We believe that conducting adversarial training on some highly challenging datasets may further enhance the effectiveness of SPC on AIME2024, which can be explored in the future.\n\n**W3. Further uses of SPC**\n\nWe thank the reviewer for this insightful suggestion. We agree that the utility of SPC extends beyond resampling, such as refinement. To explore, we conducted a simple experiment. When our SPC critic identifies an incorrect reasoning step, we provide the solver with the previous erroneous step and the critic's feedback, guiding it to perform a targeted refinement (up to five attempts). Interestingly, we found that the refinement performance of Llama-3.1-8B-Instruct on MATH500 reached 63.4%, which is slightly higher than the 62.8% achieved with re-generation. We hypothesize that the moderate gain is due to the inherent difficulty of the solver performing the refinement task itself; the solver must recognize and overcome its initial flawed reasoning, which can \"anchor\" its subsequent attempts. This aligns with recent studies on the challenges of self-correction [1,2].\n\nCrucially, this inspires a compelling future direction to employ our SPC method in improving models' refinement ability. This would involve adapting the data sampling process to a refinement process during RL data collection. We believe this could be a very interesting direction for future work.\n\n[1] Kumar et al. \"Training Language Models to Self-Correct via Reinforcement Learning.\" ICLR 2025.\n\n[2] Kamoi, Ryo, et al. \"When can LLMs actually correct their own mistakes? a critical survey of self-correction of llms.\" TACL 2024.\n\n**Other comments - Discussion on GANs**\n\nSome early GANs [3, 4] achieve great success in image generation through adversarial training, but it is difficult to transfer this approach to NLP tasks. This is because images are continuous signals, which makes it easier for GANs to optimize the generator, allowing the generated images to gradually approach real images by minimizing the cross-entropy loss on the discriminator’s real/fake logits. However, for discrete tokens in text tasks, the optimization methods used by GANs are difficult to be effective.\n\nOur SPC is fundamentally different from the adversarial training framework of GANs on vision tasks. For text tasks with discrete tokens, we leverage reinforcement learning to achieve adversarial learning. Here, the sneaky generator’s goal in RL is not only to attack the critic, but also to generate more subtle errors. We finally obtain a step-level critic for assessing LLM reasoning and providing valuable feedback. This is fundamentally different from the mutual improvement framework of adversarial training in GANs.\n\n[3] Goodfellow et al. \"Generative Adversarial Nets.\" NeurIPS 2014.\n\n[4] Karras et al. \"A Style-Based Generator Architecture for Generative Adversarial Networks.\" CVPR 2019."}]

Author rebuttal:

Meta-review: The paper introduces Self-Play Critic (SPC), a novel framework that trains a critic model to evaluate step-by-step LLM reasoning. The core contribution is an adversarial self-play game where a "sneaky generator" model learns to produce subtle reasoning errors, and a "critic" model evolves its ability to detect them. This process uses reinforcement learning from game outcomes, which reduces the need for extensive, step-level human annotations.

On the positive side, the reviewers found that the paper presents an original and well-motivated approach to a significant problem. Reviewers (czJQ, JDJJ, oHJP) described the idea as novel, intuitive, and a useful contribution. They noted that the framework effectively alleviates the data acquisition bottleneck for process reward models by generating its own challenging training data. The reviewers also found the experimental results to be solid and comprehensive, showing that SPC consistently improves performance across multiple benchmarks and outperforms baselines.

The reviewers identified areas of improvement, primarily concerning the scope and generalizability of the proposed method. One reviewer (pEmR) noted that the experiments were confined to the domain of mathematical reasoning, which raises questions about whether the "sneaky generator" strategy would be effective for more diverse tasks like coding or assistant tasks. Another reviewer (JDJJ) initially questioned if the benefits were specific to the Qwen model architecture, though this concern was later addressed by the authors with new experiments. Other points included the dependency on an initial human-annotated dataset to start the process and an initial lack of detail on the computational cost of the training loop. I think this is an important point to make more clear in the paper, since the initial annotated dataset deviates from the "Self-Play" claims in the abstract.

Based on these reviews, I recommend accepting this paper. The reviewers were excited about the paper's novel and intuitive framework for improving reasoning critics without extensive human annotation. They highlighted the originality of the step-level self-play setup, its ability to address the data acquisition bottleneck, and its strong empirical performance on several challenging benchmarks. The authors were responsive during the discussion period and provided additional experiments on a different model family (Llama-3.1), which resolved the concerns led to raised their scores. I expect the authors to incorporate the detailed feedback from the reviewers into the final version of the paper, especially the new results, and some clarification on the use of annotated data in addition to self-play.
Decision: Accept (poster)