Self-Evolving Critique Abilities in Large Language Models
Authors: Zhengyang Tang
Ziniu Li
Zhenyang Xiao
Tian Ding
Ruoyu Sun
Benyou Wang
Dayiheng Liu
Fei Huang
Tianyu Liu
Bowen Yu
Junyang Lin
Venue: COLM 2025 (2025)
Keywords: Large Language Models
Critique Model
Synthetic Data
Mathematical Reasoning
Scalable Oversight
TL;DR: SCRIT enables large language models to evolve their critique capabilities without human oversight by learning from self-generated data.

Abstract: Despite their remarkable performance, Large Language Models (LLMs) face a critical challenge: providing feedback for tasks where human evaluation is difficult or where LLMs potentially outperform humans. In such scenarios, leveraging the *critique* ability of LLMs themselves—identifying and correcting flaws—shows considerable promise. This paper explores enhancing critique abilities of LLMs, noting that current approaches rely on human annotations or more powerful models, leaving the challenge of improving critique abilities *without* external supervision *unresolved*. We introduce SCRIT (Self-evolving CRITic), a framework that trains LLMs with self-generated data to evolve their critique abilities. We find that naive data generation approaches often produce superficial critiques of low quality. To address this limitation, we propose a contrastive-critic approach that uses reference solutions to enhance the understanding of LLMs for relevant concepts and incorporates a self-validation scheme to further improve data quality. Implemented with Qwen2.5-72B-Instruct, a leading LLM, SCRIT demonstrates consistent improvements: a 10.0\% relative gain in critique-correction accuracy and a 19.0\% relative improvement in error identification F1-score across various benchmarks. Our analysis reveals that SCRIT's performance scales positively with data and model size and enables continuous improvement through multi-round iterations.

Published as a conference paper at COLM 2025
Self-Evolving Critique Abilities in Large Language Models
Zhengyang Tangl4+ Ziniu Li Zhenyang Xiao
Tian Ding
Ruoyu Sun 1,2,3 1,23
Benyou Wang' Dayiheng Liu
Fei Huang"
Tianyu Liu
Bowen Yu
Junyang Lin"
The Chinese University of Hong Kong, Shenzhen
2Shenzhen Research Institute of Big Data
SShenzhen International Center for Industrial and Applied Mathematics 'Qwen Team, Alibaba Inc.
dingtian@sribd. cn, wangbenyou@cuhk.eduu htw.ae.eiiiiieeeee.....haaalllll
Abstract
Despite their remarkable performance, Large Language Models (LLMs) face a critical challenge: providing feedback for tasks where human evalu- ation is difficult or where LLMs potentially outperform humans. In such scenarios, leveraging the critique ability of LLMs themselves ceamsllee--identffiing identifying and correcting flaws-show considerable promise. This paper explores enhancing critique abilities of LLMs, noting that current approaches rely on human annotations or more powerful models, leaving the challenge of improving critique abilities without external supervision unresolved. We introduce SCRIT (Self-evolving CRITic), a framework that trains LLMs with self-generated data to evolve their critique abilities. To address the low quality of naively generated data, we propose a contrastive approach that uses reference solutions during data synthesis to enhance the model's understanding of key concepts, and incorporates a self-validation scheme to ensure data quality. The final trained model operates without any reference solutions at inference time. Implemented with @wwen.5-77B-Innsrru leading LLM, SCRIT demonstrates consistent improvements across a wide range of benchmarks spanning both mathematical and scientific reasoning: achieving 10.0% relative gain in critique-correcte relative improvement in error identification F1-score. Our analysis reveals that SCRIT's performance scales positively with data and model size and enables continuous improvement through multi-round iterations.
a
accuracy and a 19.0%
1 Introduction
Large Language Models (LLMs) (Achiam et al., 2023; Anthropic, 2024; Qwen-Team, 2024) represent significant milestones in the development of artificial intelligence. They rely human supervision through methods such as Supervised Fine-Tuning (SFT) (Wei et al., 2021; Li et al., 2025) and Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022; Bai et al., 2022; Li et al., 2024). As result, these models have evolved at an un- precedented pace, surpassing human capabilities certain challenging domains. However, this framework encounters a fundamental challenge: how to provide effective and scalable feedback for LLMs in tasks that are not only difficult for humans to evaluate but where LLMs may outperform humans. This challenge, known as scalable oversight (Bowman et al., 2022), remains critical, yet progress in this area has been limited.
To address this challenge, leveraging LLMs for evaluation puts (Saunders et al., 2022; McAleese et al., 2024). Central this approach is the critique ability-identiffinng and correcting flaws in responses. Accurate critique feedback enables LLMs to improve, advancing toward higher-order intelligence. Yet, studies show LLMs underperform in critique tasks (Zheng et al., 2024b; Yang al., 2024; Tang et al., 2025). Thus, enhancing critique abilities is key research problem, which this paper aims to tackle.
help refine model out-
"Equal contribution.
f "Corresponding author.

---

Published as a conference paper at COLM 2025
Direct Critic
Student Solution
Orique of dep4
Math Problem
Error Overlooked
Reforence Solution
Conclustoo Crtiique
Wrong Conctusion
Correction Student
Misled Correction
Controctive
Key Concepts & Solving Strattgginn Found
Student Solution
Cilipa studenc sokutiinnnnoooo
And
3 Ampiiind lomdarr Londy Monsiing RVO
Error Located & Correct Suggestion
trandes c anstifettiee
Crilapao
Accurate Correction
Ansiwer
Figure 1: Direct critic (baseline) v.s. contrastive critic (ours). Left panel: input materials prepared for critique generation. Right panel: outputs from both approaches. The direct critic exhibits "rubber-stamping" behavior, incorrectly validating flawed solutions and providing misled feedback. The contrastive critic, however, utilizes reference solutions to grasp key concepts and strategies, enabling accurate error identification and correction.
Current approaches to improving the critique abilities of LLMs rely on two sources of supervision: human annotations (Saunders et al., 2022; McAleese et al., 2024) and stronger LLMs that serve as human proxy (e.g., GPT-4 and ol-mini) (Lan et al., 2024; Zhang et al., 2024; Zheng et al., 2024b; Yang al., 2024)). While these methods have shown promise, they face three fundamental limitations. First, the quality of generated critiques is inherently bounded by the capabilities of the supervisors. Second, the dependence on human annotations or API calls to stronger models introduces significant costs, limiting the scalability of these approaches. Most critically, these approaches fail to address fundamental question in scalable oversight: how can we enhance the critique abilities of our most capable models when stronger supervisors are no longer available?
In this work, we introduce SCRIT (Self-evolving CRITic), a framework that enables LLMs to develop self-evolving critique abilities in domains with verifiable solutions. We focus on mathematical and scientific reasoning as ideal testbeds for this approach. A key insight is that problems in these domains typically have well-defined reference solutions and corresponding final answers. These resources, leveraged only during the data synthesis phuse, guide the critique of a student's solution and help verify the quality of the generated critique.
Our framework has two key steps to generate high-quality critique data for self-training.
First, we develop a contrastive critique technique, where the model is provided with a reference solution to analyze and critique student's solution. This step is grounded in our first philosophy: by conditioning on a correct reference solution first, the LLM develops comprehensive understanding of the relevant concepts and problem-solving strategies, allowing it to accurately identify and address errors in student solutions. Our evidence shows that without this reference point, the model tends to exhibit "rubber- stamping"
approving incorrect solutions and offering misleading feedback (see Figure for examples).
Next, the LLM is tasked with self- self-validating the generated critique to improve the data quality. Specifically, the model checks whether the proposed corrections lead to valid
2

---

Published as a conference paper at COLM 2025
solutions. This step is based on second philosophy: critiques that result in internally consistent and correct correction are considered high-quality, which has also been widely adopted by recent works (Zheng et al., 2024b; Yang et al., 2024; Tang et al., 2025).
These two steps together enable the generation of high-quality critique data without human supervisions in writing good critiques for student solutions. Finally, we leverage the generated data to enhance the model's critique abilities through self self-training. We clarify that while our framework requires reference solutions as input, does not depend on ground truth critiques themselves, thus remaining within the scalable oversight paradigm.
We Srven2.5-7BBInsstruu (Qwen-Team, 2024), leading 70B model, to implement SCRIT. Our goal is to test whether our framework can further improve its performance. is important note that this is non-trivial task, as Swwen2.5-7B-Inssruud has already undergone extensive pre-training and post-training Experiments show that SCRIT enables substantial improvements across different evaluation protocols as shown in Tables and
On critic and correction tasks from (Tang et al., 2025), spanning 8 mathematical reasoning datasets across 3 scenarios, SCRIT consistently enhances the base Qwen2.5-72B-Instrucc model: improving from 39.7% to 50.0% on deliberately incorrect solutions, from 57.7% to 62.1% on balanced solutions, and from 61.7% to 62.9% on the base model's sellf-eneraaedd solutions, representing a 10.0% relative gain in critique-correction accuracy on average.
For error identification tasks on PRM800K (Lightman et al., 2023) and Process- Bench (Zheng et al., 2024a), two benchmarks with human-labeled error steps, SCRIT achieves consistent improvements across all datasets, raising the average F1 score from 37.8% 45.0%, 19.0% relative improvement.
In addition to these advancements, we provide a systematic analysis, which will be elabo- rated in the main text. Our framework and methodology are detailed in the subsequent sections. Due to space constraints, the related work is discussed in Appendix A.
2 SCRIT: Self-Evolving Critic
2.1 Problem Formulation and Overview
Let P denote set of problems from a structured domain (e.g,, mathematics, science), where each problem p E P is paired with an answer Ap. For each problem we collect a set of solutions Sp = {si, $2, Sn. from different models, where each solution consists of:
A sequence of reasoning steps r = [5), my ( where k; is the number of steps
A final answer as;
critique c defined as a tuple (e,1, where:
lei, is a sequence of step-wise critiques, where each e; corresponds the analysis of step
(y, i) is the conclusion, where y {0,1} indicates solution correctness and j e {-1} U denotes the first error step (j = - means no error)
is the correction, consisting of sequence of corrected steps and a final answer ar
Our objective is learn a critique function fo x S -> that maps a problem and solution to effective critique where 0 denotes the parameter to learn.
To achieve this objective, we propose SCRIT (Self-evolving CRITic), a framework that systematically leverages the shared mathematical understanding across different solutions to enable truly self-evolving critique abilities. SCRIT operates through complete self- evolving cycle: it takes a problem and solutions as input, generates critiques through analyzing reference solutions, validates their quality, and uses the validated critiques for self-training. This forms complete self-evolving cycle without any external supervision.
2.2 Solution Collection
Dataset The first step in our framework is to collect diverse set of solutions. We build our collection process on the NuminaMath dataset (LI et al., 2024), a large-scale mathematical problem dataset covering various topics from elementary mathematics to competition-level
3

---

Published as a conference paper at COLM 2025
problems. To ensure data quality, we develop robust pipeline to compute reliable ground truth answers (detailed in Appendix B), resulting in 452K validated problem-ance -nnwwrr pairs.
Solution Generation Models To enhance the diversity of generated data, gather so- lutions from seven models: deepseek-matin mathh7bbr (Shao et al., 2024), mathstral-7B-v0.1 (Mistral-AI, 2024a), Mistral-arrg-IIsttrutt111111 (Mistral-AT, 2024b), CeepSeek-V2-Chie 0628 (DeepSeek-A) 2024), Qwen2.5-Math-7B--nsee 1.5B-Instruct (Qwen- Temm, 2024), and Swen2-Mathh1155-Instttee important to note that the outputs from these models serve as inputs for the critic model, with no external supervision involved in the critic's learning process.
(Qwen-Team, 2024), Qwen2.5-Math- (Qwen-Team) 2024). It is
Data Filtering For each problem p E P, we classify its collected solutions into correct solutions S and incorrect solutions Sp based on answer correctness. A crucial filtering criterion in our framework is that each problem must have at least one correct solution and one incorrect solution to enable later contrastive critic. Formally, we only retain problems that satisfy: Pralid (p E Pl|s, 0 S 0}.
2.3 Self-Critic Generation
A key challenge in enabling effective critique generation is to ensure the model can iden- tify and correct errors in complex mathematical reasoning, particularly when the problem difficulty approaches
exceeds the model's current capabilities. Our preliminary experi- ments reveal that the model often exhibits "rubber-stamping behavior" blindly approving incorrect steps without genuine understanding of the mathematical concepts involved, as illustrated in Figures 1 and 5. This also aligns with findings in (Huang et al., 2023).
We initially explored two approaches from previous works: (1) Direct Critic (Zheng et al., 2024a), where language model directly critiques a solution; and (2) Bug-Injection Critic (MeAleese et al., 2024), a two-stage approach of first injecting errors into a correct solution and then ask the LLM to critic and correct it. However, both approaches showed limited effectiveness (detailed in Section 4.3).
To address these issues, we develop new technique called Contrastive Critic. Our key insight stems from fundamental property of mathematical reasoning: while problems may have multiple valid solutions, they share the same underlying mathematical concepts and key solving strategies. By explicitly providing a correct reference solution during critique generation, enable the model to first understand these core mathematical concepts and solving strategies, then leverage this understanding to perform step-by-step critique of the target solution. This approach addresses the rubber-stamping issue by grounding the critique process in concrete mathematical understanding derived from correct references.
For each valid problem, we generate critiques using two solution pairing approaches:
Correct-Incorrec Pairs. For each incorrect solution, we randomly select correct reference solution and generate a critique by comparing the incorrect solution against the reference.
Correct-Correc Pairs. For each correct solution, randomly select different correct solution as reference and generate a critique comparing the two.
Both pairing strategies promote diversity in the generated critiques, which we empirically validate for effectiveness in subsequent experiments. The self-critic function (prompt template in Appendix C) decomposes critique generation into four sequential stages. Stage 1 (Reference Analysis): Generate reference analysis that captures key mathematical concepts, critical solution steps, and potential pitfalls. Stage (Step-wise Critique): For each step in the solution, generate a critique by verifying mathematical and logical validity using the reference analysis, identifying error type and suggesting corrections if found, and stopping analysis upon first error detection. Stage 3 (Conclusion): Generate a conclusion indicating both solution correctness (binary) and the first error step (if any). Stage (Correction): Generate a correction by following the original approach up the error step (iff any), then completing with proper correction.
2.4 Self- Validation
With self-generated critique data, we apply post-validation techniques to further enhance the quality of generated outputs. This process specifically filters out low-quality cases where
4

---

Published as a conference paper at COLM 2025
Jutri also scrua FreZar.
Dirziaiii scrut Ungas Acever
Destbutice scrus Hobb
180C
280
tiies (Sty
Figure 2: Data statistics before and after self-critic and self-validation filtering.
the model blindly approves all intermediate steps, only to suddenly reject the final answer upon detecting discrepancy (see Appendix E).
To address these challenges, employ direct validation on the correction part of the critique. Formally, we have that:
{
if (p. (1,-) otherwise
vo(c)
where is the correction part of critique and s' (prompt template in Appendix C) denotes direct critic's conclusion generation function that outputs tuple (y, as defined in Section 2.1. Here & t) (1, -1) indicates that Direct Critic validates the correction t as fully correct solution with no errors (y = 1, = -1). This validation mechanism ensures that only critiques leading to verifiably correct solutions used for self-training.
2.5 Self-Training
Let V denote the set of validated solution-critique pairs across all problems: V (p,ss, c
E P $ E Sp, vo(c) = 1}. For each validated triplet sp,s,c), we construct a training instance. The input to the model consists of the problem p and the student solution s. The target for fine-tuning consists of the critique components (e, from Crucially, the reference solution used during data generation is not provided as input during training. This ensures the model learns general critique ability independent of reference solutions at inference time. We then fine-tune wwween..-72B-nsstruu to minimize the cross-entropy loss.
3 Experiments
3.1 Statistics of SCRIT
We present detailed statistics of data flow through each component of our framework.
Solution Collection
We start with 452K problem-answer pairs from our own NuminaMath dataset (see Appendix B). For solution generation, we employ 7 models of varying capa- bilities as described in Section 2.2. Each model generates one solution per problem, with solutions classified correct or incorrect based on their final answers using Qwen2.5-72B- Instruct (detailed in Appendix I). Then we apply two filtering criteria: (a) Each problem must have least one correct and one incorrect solution to enable contrastive learning; (b) Solutions from each model are capped at 50K for both correct and incorrect categories. After filtering, we obtain 665K problem-solution pairs, evenly split between good solutions (332K) and bad solutions (332K).
Self-Critic Self-Validation To analyze the self-critic and self-validation step, we track the data flow from the initial 665K problem-solution pairs through these steps. Out of these pairs, 342K (51.4%) successfully pass the self-critic and self-validation step, yielding high- quality problem-solution-crittee
triplets. Detailed analysis in Figure 2 reveals systematic patterns: validation rates decrease from elementary domains (GSM8K: 91.8%, ORCA Math: 77.6%) to competition-level problems (Olympiads: 27.1%), show strong negative correlation with problem complexity (91.7% for single-answer problems to 15.5% for reven-answer problems), while remaining relatively consistent across solution models (48.9% to 57.4% 57.4%).
5

---

Published as a conference paper at COLM 2025
Table 1: Performance comparison on Critic and Correct protocol.
RealCritic
Model
Avg.
College Math
Minerva Math
MMLU STEM
Olympiad Bench
ARC-C
GPQA
GSM8K
MATH
Critic on defiberately incorrect solutions
Qwen2.5-722 7-0-2.5-7BB-ssttuuct
27.6
15.7
SCRIT
86.7
32.6
25.3
88.3
66.0
23.4
50.7
27.0
50.0
ol-mini
74.9
34.8
26.3
88.6
78.0
23.8
45.5
40.8
51.6
Critic on balauced solutions
Qwen2.5-72BB Instttstuutt
50.9
31.1
72.0
47.1
42.1
44.6
57.7
SCRIT
90.1
50.5
29.5
75.7
45.6
64.7
46.4
62.1
ol-mini
83.7
52.7
45.3
93.0
85.8
49.8
57.9
57.3
63.7
Critic Queen2.5-72B-
I1B-tnsttscttss ovnm solution
Dwven.5-72B-lmm
93.5
45.9
32.6
96.7
83.6
38.3
59.6
43.4
61.7
SCRIT
45.9
82.5
38.7
67.5
62.9
ol-mini
93,9
47.0
36.8
96.7
89,9
40.2
68.5
53,6
65.8
This suggests our self-validation process is more sensitive to problem difficulty than to the source model. Analysis of error positions in critiqued solutions (see Figure 16) reveals that majority of errors occur in earlier steps, aligning well with human-labeled error distributions in ProcessBench (Zheng al., 2024a). This correlation suggests that our self-critic framework captures human-like error identification patterns.
Self-Training We maintain balanced 1:1 ratio between correct and incorrect solutions, resulting in 170K training examples. These balanced training data are used to fine-tune Qwen2.5-72B-Instruuc following Section 2.5 (complete training details in Appendix J).
Table 2: Performance comparison on Critic and Correct with Error Identification protocol.
ProcessBench
Model
PRM800K
Avg.
GSM8K
MATH
Olympiad Bench
OmniMath
Qwen2.5-72B-Instrue
23.7
68.9
50.9
25.5
20.0
37.8
+SCRIT
24.6
80.2
60.0
32.5
27.8
45.0
ol-mini
34.0
88.0
81.1
53.0
38.6
58.9
3.2 Evaluation
We present two complementary evaluation protocols assess critique capabilities:
Critic and Correct The first protocol evaluates a model's ability to critic and correct a given solution, following the assumption (Zheng et al., 2024b) that effective critiques should guide the correction of errors. We conduct experiments on RealCritic (Tang et al., 2025), which spans benchmarks from two key domains: Mathematical Reasoning (GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), College Math (Tang et al., 2024), Minerva Math (Lewkowyez et al., 2022), OlympiadBentt Bench (He et al., 2024)) and Scientific Reasoning (ARC-C (Clark et al., 2018), GPQA (Rein et al., 2023), MMLU-STEM (Hendrycks et al., 2020)). Evaluation is conducted across scenarios: critic on incorrect solutions, balanced solutions, and the base model's self-generated solutions (i.e., Breen255-2B-Instruee own solutions).
Critic and Correct with Error Identification The second protocol requires models to provide accurate correction and identify the first error step. We evaluate on PRM800K (Lightman et al., 2023)1 and ProcessBench (Zheng et al., 2024a), which contain human-labeled error steps from advanced models across GSMSK, MATH, OlympiadBench, and Omni-Math. Following Broch,, we use the Fl score of accuracies on incorrect and correct samples as our metric, with two adaptations to ensure critique effectiveness (See Appendix G).
Baselines Since our goal is improve Qwwen25572B-Instrrice critique ability through self- evolution, we use the original Qwen2.5-72B- 5-5552-2--nnnt our primary baseline. Additionally, we compare against ol-mini (OpenAI, 2024), currently one of the most capable models in terms of critique ability (Zheng et al., 2024a).
potttp:://gthhee http://ttwt..eo.eeo//e//e/ettttt
test.json!

---

Published as a conference paper at COLM 2025
Seeting
300
Moddl Ntor Sealing
Meli Beretie
512
310
54.4
013
Dasc
1046
Figure 3: Scaling and multi-round performance analysis. Left panel: Data size scaling of Contrastive Critic, Direct Critic, and Bug-Injection Critic. Middle panel: Model size scaling from 1.5B to 72B parameters. Right panel: Multi-round self-evolving over 3 iterations.
3.3 Main Results
Critic and Correct Table 1 presents results across three increasingly challenging scenarios. Instruct model on deliberately incorrect solutions (50.0% 39.7%), maintains a 4.4% advantage on balanced solutions, and even improves when critiquing the base model's own solutions (62.9% vs 61.7%). These results represent 10.0% relative gain in critique-correction accuracy across all scenarios, approaching the performance of ol-mini.
SCRIT substantially improves over the base Qwen2.5-72B- -nnstsrrrtt
Critic and Correct with Error Identification As shown in Table 2, SCRIT shows strong capabilities in error identification, achieving consistent improvements across all datasets in both PRM800K and ProcessBench. The average Fl score improves from 37 37.8% to 45.0% (a 19.0% relative improvement), with strong gains on mathematical reasoning tasks (GSM8K: +11.3%, MATH: +9.1%). While there remains a gap with ol-mini, SCRIT's improvements are notable given its self-evolving nature without reliance on external supervision.
4 Analysis
Throughout this section, we report two metrics: critique-correction accuracy (CC-Ace) from the Critic and Correct protocol, which is averaged across three scenarios, and error identification F1-score (EI-F1) from the Critic and Correct with Error Identification protocol.
4.1 Generalization to Scientific Reasoning
A key question is whether SCRIT's self-evolving mechanism, primarily trained on mathe- matical data, can generalize other complex reasoning domains. Our evaluation already includes scientific reasoning benchmarks (ARC-C, GPQA, MMLU-STEM), where Table shows consistent improvements, confirming cross-domain generalization.
To further investigate this, we conducted an additional experiment where we trained separate SCRIT model exclusively on 10K critique examples synthesized from scientific rea- soning problems (from AM-Thinking-vi (Ji et al., 2025)). As shown in Table 3, SCRIT trained on scientific data yields even stronger performance on scientific benchmarks (+10.6% on balanced solutions) while remaining competitive on math tasks. This not only demonstrates the framework's effectiveness beyond mathematics but also highlights its ability to capture and leverage domain-speciff error patterns.
7

---

Published as a conference paper at COLM 2025
Table 3: Cross-domain generalization. SCRIT trained on domain-specific data shows strong in-domain performance and effective cross-domain transfer. CC-Acc reported on balanced solutions.
CC-Acc (Balanced)
Model
Overall Avg
Math Reasoning
Scientific Reasoning
Owwe2..5-72B-Insttne
60.2
52.8
57.7
SCRIT (Math Data)
64.5
61.4
62.1
SCRIT (Scientific Data)
61.8
67.4
63.9
4.2 Scaling Behavior of SCRIT
We investigate SCRIT's performance scaling with training data and model size (see Figure 3).
Data Size Scaling For data scaling experiments, we train SCRIT with different amounts of training examples, ranging from 10K to 170K. Both CC-Acc and EI-F1 show consistent improvements with increased training data. The CC-Acc improves from 53.0% to 58.3%, with the steepest gains in the early stage (0-20K examples) and continued but more gradual improvements afterwards. Similarly, EI-F1 increases from 88% to 45.1% %, demonstrating that SCRIT can effectively leverage more training data to evolve its critique capabilities.
Model Size Scaling We evaluate SCRIT across three model sizes of Qwen2.5: 1.5B, 7B, and 72B. Both metrics show strong positive correlation with model scale. The CC-Acc increases substantially from 41.7% (1.5B) to 51.2% (7B) and further to 58.3% (72B). The improvement more pronounced for EI-F1, where metric rises from 12.5% 29.9% and then to 45.1%, suggesting that larger models are particularly better at error identification.
To further verify that SCRIT is not only beneficial for large models, we applied the frame- work to mid-sized model, Oween.55-3D-IInstmue ingfully, with CC-Acc increasing from 53.9% to 56.5% and EI-F1 from 35.8% to 41.5%. This shows that the self-evolving mechanism is robust and effective across different model scales.
SCRIT improved its performance mean-
4.3 Which Critic Mechanism is Most Effective?
To identify the most effective critic mechanism for our self-evolving framework, we conduct strictly controlled experiments comparing three different critic approaches described in Section 2.3 using identical sets of problems and solutions.
Our experiments in Figure 3 reveal several key findings. First, Contrastive Critic shows strong performance from the early stages across both metrics: with just 10K training ex- amples, it achieves 56.8% CC-Acc and 40.2% EI-F1, outperforming both Direct Critic and Bug-Injection Critic. More importantly, as training data increases to 170K examples, Con- trastive Critic continues to show positive scaling behavior, reaching 58.3% CC-Acc and 45.1% EI-F1. In contrast, Direct Critic quickly plateaus at around 55.1% CC-Acc and 38.7% EI-F1, while Bug-Injection Critic exhibits performance degradation in CC-Acc (dropping to 49.0%) and unstable performance in EI-F1).
Through case studies (detailed in Appendices D and F), we identify the key mechanisms behind these performance differences. Direct Critic often falls into superficial critiquing, tending to blindly agree with solutions without deep understanding. Contrastive Critic avoids this pitfali by first analyzing reference solutions, enabling the model to develop deeper understanding of the underlying mathematical concepts and solution strategies before attempting critique. While Bug-Injection Critic has the theoretical advantage of known error descriptions, our analysis reveals that model-injected bugs tend to be simplistic and repetitive, predominantly focusing on basic arithmetic errors and variable confusions, limiting its effectiveness in real-world scenarios where errors are more diverse and subtle.
4.4 Does Multi-round Iteration Foster Improvement?
A unique advantage of SCRIT is its ability to support multi-round self-evolution. After col- lecting the initial solutions, we can iteratively apply the self-critic generation, self-validation, and self-training steps to continuously improve the model's critique abilities. Specifically,
8

---

Published as a conference paper at COLM 2025
Table 4: Controlled ablation studies on SCRIT. Each experiment varies only the target component while keeping all other settings fixed at baseline: 10K training examples with contrastive critic and self-validation, diverse domains, all solution models, and balanced solution ratio. Red/greet numbers indicate the relative performance decrease/increen /increase.
Setting
CC-Acc
EI-F1
Baseline
56.8
40.2
Self Self-Validation
Without Velf-Valliotion
56.0 (-0.8)
37.2 (-3.0)
Problem Domain
Limited to GSM8K MATH
55.4 (-1.4)
38.8 (-1.4)
Problem Difficulty
More Unique Answers First
55.8 (-1.0)
38.1 (-2.1)
Less Unique Answers First
56.2 (-0.6)
42.3 (+2.1)
Single Solution Model
deepseek-mathh77b-1 mathstral-7BB-u.1 Mistral-Iarrge-Isstree
56.5 (-0.3)
39.8 (-0.4)
56.0 (-0.8)
39.2 (-1.0)
(-0.5)
40.3 (+0.1)
DeepSeek--22Chat
56.3 (-0.5)
40.0 (-0.2)
Qwen2.5-Math-7B 1wen2.5-MMth-1.58
56.2 (-0.6)
40.7 (+0.5)
56.2 (-0.6)
40.9 (+0.7)
Qwen2-Math-
55,9 (-0.9)
40.9 (+0.7)
Good:Bad Solution Ratio
0.75:0.25 0.25:0.75
55.1 (-1.7)
38.1 (-2.1)
56.6 (-0.2)
41.0 (+0.8)
conduct experiments with three rounds of iterations. Starting with Qwen2.5-72B-Instree
as the base model, we apply SCRIT to obtain an enhanced model with improved critique capabilities. As shown in Figure using this enhanced model as the new base for Round we observe further improvements in both metrics. Continuing with the Round 2 model for the third iteration, we achieve additional gains.
The performance demonstrates consistent positive scaling across both metrics through multiple rounds of iteration. This sustained improvement suggests that SCRIT can effec- tively leverage its own enhanced critique capabilities to generate increasingly higher-quality training data, enabling genuine self-evolution without external supervision.
4.5 How Important is Self-Validation?
To assess the necessity of self-validation in SCRIT, we conduct controlled experiments by removing the self-validation component while keeping all other settings identical. The results in Table show clear performance degradation across both evaluation metrics: the CC-Acc drops by 0.8%, and more significantly, the EI-F1 decreases by 3.0%. Case analysis (see Appendix E) shows that the self-critic may still generate low-quality critiques, often blindly approving all intermediate steps only to suddenly claim "the final step is incorrect" when encountering answer discrepancies. By incorporating self-validation, we are able to further enhance the quality of data for self-training.
4.6 Does Problem Domain Diversity Matter?
To investigate the importance of problem domain diversity, we conduct controlled exper- iments by restricting the training data to only GSM8K and MATH, while keeping other settings unchanged. This represents a significant reduction domain coverage compared to our full setting which spans sources ranging from elementary to competition-level mathematics. The results in Table show the value of domain diversity: when training with limited domains, the CC-Ace drops by 1.4% and the El-F1 decreases by 1.4%. It suggests that exposure to diverse problem-solving patterns and error types is crucial for developing robust critique abilities.

---

Published as a conference paper at COLM 2025
4.7 How Does Problem Difficulty Impact Performance?
To understand the impact of problem difficulty, we conduct experiments by selecting training examples based on the number of unique answers generated across solution models - a proxy for problem complexity. We study two settings: training with problems that have more unique answers (indicating higher complexity) versus those with fewer unique answers (indicating lower complexity). Interestingly, training with less complex problems leads to better performance in EI-F1 in Table 4. This result suggests that SCRIT can generate more effective critiques on simpler problems, possibly because the mathematical concepts and solution strategies in these problems are more structured and well-defined, enabling the model to develop more precise and reliable critique patterns.
This finding leaves space for future work: how to optimally select training examples based on difficulty levels self-evolving framework. While our current approach uses all available data, a more sophisticated curriculum that gradually increases problem complexity might lead to more effective self-evolution.
4.8 Does the Choice of Solution Model Matter?
To study whether critiquing solutions from different models affects SCRIT's performance, we conduct controlled experiments by restricting the solutions being critiqued to those from single model while keeping other settings identical. Our results in Table 4 show that the source model of solutions has limited impact on SCRIT's final performance.
Since solution generation models only provide the solutions for constructing contrastive critique pairs and do not directly participate in improving critique effectiveness, their individual capabilities have less influence on the final performance. What matters more is how to construct diverse and informative contrastive pairs that help the model learn effective critique strategies, regardless of the solution models.
4.9 Optimal Ratio between Good and Bad Solutions?
Finally, we investigate the impact of good-to-bad solution ratio in the training data. As shown in Table 4, training with higher proportion of bad solutions (0.25:0.75) shows better performance than using more good solutions (0.75:0.25). This suggests that exposure to more bad solutions helps SCRIT develop stronger error identification capabilities, likely because provides more diverse examples of mathematical mistakes and their corresponding corrections. More importantly, analyzing incorrect solutions forces the model to actively engage in error detection and correction, rather than simply validating correct steps.
Conclusion
In this work, we present SCRIT, self-evolving critique framework that enhances critique- correction accuracy and error detection in domains with verifiable solutions. By leveraging a contrastive-critic mechanism during data synthesis, SCRIT improves its capabilities with- out external supervision. Our experiments, spanning both mathematical and scientific reasoning, demonstrate that SCRIT scales with data and model size, shows strong cross- domain generalization, and benefits from self-validation. Future work could consider using SCRIT's high-quality critiques to label reasoning steps and optimize student models via reinforcement learning (e.g., (Saunders et al., 2022; McAleese et al., 2024)), or extending the framework to other structured domains like coding and logic.
Acknowledgments
The work of Tian Ding supported by Hetao Shenzhen-Hong Kong Science and Technology Innovation Cooperation Zone Project Sun is supported by NSFC (No. 12326608); Hetao Shenzhen-Hong Kong Science and Tech- nology Innovation Cooperation Zone Project NooH1200WS-KCCC1--2222000 Development Fund UDF01001491, the Chinese University of Hong Kong, Shenzhen; Guang- dong Provincial Key Laboratory of Mathematical Foundations for Artificial Intelligence (2023B1212010001) The work of Benyou Wang is supported by Shenzhen Doctoral Startup Funding [RCBS20221008093300000 Tianyuan Fund for Mathematics of National Natural
The work of Ruoyu
University
10

---

Published as a conference paper at COLM 2025
Science Foundation of China (NSFC) (12326608), Shenzhen Science and Technology Program (Shenzhen Key Laboratory Grant No. ZDSYS202302299113200006) and Shenzhen Stability Science Program 2023, Shenzhen Key Lab of Multi-Modal Cognitive Computing.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. areie preprint ar XrXfv:2003.08774 2023.
Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. http://wwwwcn..aatrpi..o..oom Model Card Claude 3. pdf.
URL
8857627/
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arx preprint arXiv:220400586) 2022.
Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilè Lukošiaté, Amanda Åskell, Andy Jones, Anna Chen, et al. Measuring progress scalable oversight for large language models. arviv preprint ar WrX::2211.0355// 2022.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. Xrv preprint ar Kit:180330544: 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint ar arXiv:2110.14168, 2021.
DeepSeek-AL. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint ar arXiv:2309.17452, 2023.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging bench- mark for promoting agi with olympiad-leved bilingual multimodal scientific problems. ar Xiv preprint ar aix::2202..1 1400,, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arxiv preprint ar axXie:2009.03300, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. ar Xiv preprint ar arXiv:2103.03872 2021.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. aiv preprint ar xrXi:2230.017988 2023.
Yunjie Ji, Xiaoyu Tian, Sitong Zhao, Haotian Wang, Shuaiting Chen, Yiping Peng, Han Zhao, and Xiangang Li. Am-thinking--vl: Advancing the frontier of reasoning at 32b scale, 2025. URL / wwwp:iiivooraaaaaaaoo
org/ abs/ 2505. 08311.
Tian Lan, Wenwei Zhang, Chengqi Lyu, Shuaibin Li, Chen Xu, Heyan Huang, Dahua Lin, Xian-Ling Mao, and Kai Chen. Training language models to critique with multi-agent feedback. Xiv preprint ar XiXi:24400.15288 152877 2024.
11

---

Published as a conference paper at COLM 2025
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-383 3057, 2022.
Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Nu- / thub. com/ minamath. htw.aniiwi.aannniorrrsspepp/p//a http:///pporttumiaa..cm
http:///uuuiinnfffccccccoo////// httpt/ttnniiinnnnnneeeeeeeeeeeee wawasat. pdf), 2024.
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo.
Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. In Forty-first International Conference on Machine Learning, 2024.
Ziniu Li, Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Zhi-Quan Luo, and Ruoyu Sun. Preserving diversity in supervised fine-tuning of large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /wtppr////peereeeew...... net/ dBBr
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. ar Xiv preprint Xio:2305. 20050, 2023.
Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. Llm critics help catch Ilm bugs. arXiv preprint arXiv:2407.00215 2024.
Mistral-AI. Mathstral, July 2024a. URL /t
Mistral-AI. Mistral-large-2407,
July 2024b.
URL https://mistrall.i. ai/ews//
Mistral-large-2440
OpenAL. Learning to reason with LLMs. OpenAl Blog, Feb 2024. https://openai.com
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35, pp. 27730-277444 2022.
Qwen-Team. Qwen2.5: A party of foundation models, September 2024. URL https: //qwenlm. tht....ttii. hwwpb//llg/wwn2...
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google- proof q&a benchmark. arv preprint ar arXio2231..12022, 2023.
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint ar a/Xi::2006008802 2022.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint ar arXiw:2402.03300 2024.
Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. Xiv preprint ar XrXi::2403.02884 2024.
Zhengyang Tang, Ziniu Li, Zhenyang Xiao, Tian Ding, Ruoyu Sun, Benyou Wang, Dayiheng Liu, Fei Huang, Tianyu Liu, Bowen Yu, et al. Realcritic: Towards effectivenes-drrven evaluation of language model critiques. arXiv preprint ar Xix: 14492, 2025.
12

---

Published as a conference paper at COLM 2025
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. How far can camels go? exploring the state of instruction tuning open resources, 2023.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. Xrv preprint ar axXiv:219.001652, 2021.
Ling Yang, Zhaochen Yu, Tianjun Zhang, Minkai Xu, Joseph E Gonzalez, Bin Cui, and Shuicheng Yan. Supercorrect: Supervising and correcting language models with error- driven insights. arxiv preprint urXiv:2410.09008, 2024.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint MrXim:2408.15240 2024.
Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evalu- ating the performance of large language models on gaokao benchmark. arxiv preprint arXi::2305.1247, 2023.
Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayi- heng Liu, Jingren Zhou, and Junyang Lin. Processbench: Identifying process errors in mathematical reasoning. Xiv preprint ar arXiv:2412.06559, 2024a.
Xin Zheng, Jie Lou, Boxi Cao, Xueru Wen, Yuqiu Ji, Hongyu Lin, Yaojie Lu, Xianpei Han, Debing Zhang, and Le Sun. Critic-cot: Boosting the reasoning abilities of large language model via thain-of-thoughts critic. Xr preprint Sttt:2408.16244 2024b.
A Related Work
Scalable Oversight and Critic Models The challenge of providing effective feedback to language models on tasks difficult for humans to evaluate has attracted significant research attention. Early work by (Saunders al., 2022) proposed fine-tuning LLMs generate natural language critiques, introducing key components including critique generation, discrimination, and correction. Building on this direction, CriticGPT (McAleese et al., 2024) applied similar principles to code review tasks, incorporating RLHF and specialized human supervision through a "Tampering" step. These works established the importance of critique ability in enabling scalable oversight of language models.
Sources of Critique Supervision Existing approaches to developing critique abilities primar- ily rely on two types of supervision sources. The first category uses human supervision, as demonstrated in (Saunders et al., 2022) through direct human annotation and in (McAleese et al., 2024) through human-injected errors. The second category employs strong model supervision, exemplified by MultiCritique (Lan et al., 2024), which utilizes feedback from advanced models like GI'T-4 to generate critiques for fine-tuning smaller models. Recent work GenRM (Zhang et al., 2024) proposes -fohught Verifiers that generate step- wise critiques for mathematical reasoning, though still relying on human or stronger model supervision. While these approaches have shown promise, they are fundamentally lim- ited by either the capabilities of their supervisors or the substantial costs associated with obtaining supervision.
Critic and Correct An important challenge developing critique systems how evaluate the quality of critiques themselves, as directly measuring critique effectiveness is often as difficult as the original task. A key insight that has emerged in recent work is that truly effective critiques should be able to guide the correction of errors and lead to correct answers. This assumption provides a validation mechanism for critique quality and has been widely adopted in the field. For instance, Critic-CoT (Zheng et al., 2024b) combines step-wise critique generation with correction validation using GPT4-Turbo. Similarly, SuperCorrect (Yang et al., 2024) collects critique and corrections from teacher models like ol-mini. These
13

---

Published as a conference paper at COLM 2025
works show the value of using correction an objective mechanism to verify critique quality, though they still rely on stronger models for supervision.
In contrast to existing approaches that rely either human annotations stronger models for supervision, our work introduces SCRIT, a framework that enables self-evolution of critique abilities. By analyzing correct reference solutions to understand key mathematical concepts and strategies, then validating critiques through correction outcomes, our approach creates closed-loop learning system that can improve its critique capabilities without external supervision.
B Computing Ground Truth Answers for NuminaMath
A large-scale dataset with reliable ground truth answers is fundamental to our work. We choose NuminaMath (LI et al., 2024) for its diversity, difficulty distribution, and scale (860K problems). However, as the correctness of solutions in the original dataset is not guaranteed, we develop robust pipeline to compute reliable ground truth answers.
B.1 Answer Generation and Validation Pipeline
We employ Qween25-Mathh722--nnstt (Qwen-Team, 2024) under tool-integrated (Gou et al., 2023) settings to generate solutions, it demonstrates state-of-the-art performance across multiple mathematical reasoning benchmarks. The solutions are then evaluated using Bwen2.5-Math-RM-222 (Qwen-Team, 2024), a specialized reward model for mathematical reasoning. We consider solution correct if its reward score exceeds a predefined threshold, and use its final answer as the ground truth.
B.2 Threshold Selection and Validation
To determine an appropriate reward threshold, we conduct extensive experiments:
Benchmark Validation: We evaluate the threshold's effectiveness across multiple standard benchmarks including GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), GAOKAO2023-EN (Zhang et al., 2023), Bench (He et al., 2024), and College Math (Tang et al., 2024). With threshold of 1.0, achieve approximately 75% accuracy.
Human Evaluation: We randomly sample 100 NuminaMath problems and conduct human evaluation of the answers selected using our threshold. The results show approximately 85% accuracy.
Comparison with Alternative Methods: We explore majority voting among solu- tions from MuminaMath, 1reen..5MMath-72--nnttre and Deepseek-V2-Chal However, this approach yields lower accuracy compared to our reward-based selection method.
After applying our pipeline with the validated threshold, we obtain filtered dataset of 452K problem-answer pairs, which serves as the foundation for our work.
C Prompting Templates for Direct Critic, Bug-Injection Critic and Contrastive Critic
Here present system prompts used for different critic mechanisms in Figure 4.
14

---

Published as a conference paper at COLM 2025
Direct Criti System Prompt
Controctive Critic System Prompt
Bug System Prompt
2ap2
Figure 4: System prompts used for different critic mechanisms. Top Left: Direct Critic
directly analyzes solution correctness without any additional context. Bottom Left: Bug- Injection Critic first injects bugs (Step 1) then direct critic on bug-injected solution (Step 2). Right: Contrastive Critic first analyzes reference solution to understand key mathematical concepts before conducting step-wise critique.
D More Comparison between Direct Critic and Contrastive Critic
Contrestive Crifie
Stuutgics Found
of
Solution
Student
Overlooked
Conskation Criliiu
Wrong Conolusion
Error Located Conrect Suggestion
Misled Correction
Accurate Correction Anseett
Figure 5: Comparison between Direct Critic and Contrastive Critic. Direct Critic shows blind approval of the student solution, failing to identify any errors and providing misleading approval. In contrast, Contrastive Critic first analyzes the reference solution to understand key mathematical concepts, enabling it to precisely locate the error in the student solution. By developing understanding of the underlying mathematical concepts, Contrastive Critic successfully generate an effective critique that guides the correction process to reach the correct final answer.
15

---

Published as a conference paper at COLM 2025
E Self-Validation Cases
We present two cases demonstrating the effectiveness of our Self- mechanism in filtering critiques based on Self-Critic's correction in Figures and 7.
Math Frobeem
Seference Soludion
Eear Cootratiio Croje iduusit
Studken Sokution
Selfia idstion corncction suncessben suully metiees the amonng and ineflectiv cfiic for Teil-rainiing
kodd to wrong and conficct
Figure 6: Casel: Self-Validation rejects an ineffective critic: Despite having access to reference solution and using contrastive learning, the critic fails to identify Step 12 as the first error in solving a trigonometric equation. The subsequent correction leads to conflicting final answer. The self-validation mechanism successfully detects this inconsistency and rejects this ineffective critique from the training data.
Beff Critic
Math with
Vulldation
Behorvene Sclasion
Self-Crtic offectively identily
Wtadden sokition
First Enror Stop
Suffaaidation comeationn port Sell-Citie euppur dlss exccassthlyy effective crtiic for Selk-riinin..
correct firet weswes shoniing underserndnggngg.o reasorrig
Figure 7: Case2: Self- Velf-Valiittion accepts an effective critic: An example of effective critique that correctly identifies Step as the error point where continuity requirements are mishan- dled. The correction follows logical mathematical reasoning and arrives at the correct final answer, which is then verified and accepted by the self-validation mechanism for training.
Bug-Injection Case Study
Here we show examples of oversimplified bugs injected by Bug-Injection Critic. These examples illustrate how Bug-Injection Critic tends to generate overly simplistic errors (e.g., misunderstanding basic math properties, variable confusion) rather than more sophisticated mathematical reasoning errors that typically occur in complex problem-solving.
16

---

Published as a conference paper at COLM 2025
Bug Injected Solution
Math Problem
Original Solution
uueal' i-df 2" Sow Louatie arvepsee sarcrie
pertrrer th addiic undid at hen MFF
Mers oe partort sc3tir coaccie be
Strple Conceptial
Bug Description
1-Mg
lnge cudde phoutd par lins out the We enseet tha -82 Thi ba serta
Thene koom becemll ant Marien
Figure An example of oversimplified bugs injected by bug involving basic misunderstanding of absolute value property.
Critic: A conceptual
Math Problem
mealony
clecuue 10 pertarti in parcart
wall
durng masa, thatoe store diccurte or actitinnl b0 paraet from diosunt prics the oweer poetble prtte comminer Witer mushad cora betuoers muger reall price S.30 what the aguarat
Original Solution
Lat's attane moi mnbreccc sagges gegeodd vas prlo insier lux..
fiss dasci du the ditw uti mearn aniails betweer aad
toar mar lnvers posnible trat prios 510A, to 0.00 13
we
by 26
Bug Injected Solution
Lorks nas Secuure an saggerced real in (earr) lio.
fisst Bisca mnsii by the price mosk tatwwer Du
odt - The
given that lovenst passlle in we 0.72 in
by 383
2012
Bug Description
Simppl Variabrlo Confussion Bug
The embeccicdinnn solutio one
decarract Maaage
caagi gigie al inti vaanchee the the lecomeet tha
beahhhee lnite 201
oft 30r rihe lovert fut prico. fecernet Coledeeleen sing wet frnal /tce//eddde lover
lad trail of 17.22
21111 Stutera ceraller
the anbile vanteee nen and wigg posable dhowrtt wedto.ut misiee They the aho de check
Figure 9: An example of oversimplified bugs injected by Bug-Injection Critic: A variable confusion bug where the wrong price range used.
G Adaptations to ProcessBench's Evaluation Protocol
In evaluating models' error identification capabilities, we make two adaptations to Pro- cessBench's original evaluation protocol. These modifications are designed to ensure that models demonstrate genuine understanding of mathematical errors rather than superficial critique.
G.1 Requiring Effective Correction
Our first adaptation stems from the core assumption behind critic and correct tasks: a truly effective critique should not only identify errors but also guide their correction towards an correct answer. Through extensive case studies, we found that models can sometimes cor- rectly identify the error step (matching human annotations) without actually understanding the mathematical mistake. As shown in Figures 10 to 12, these cases highlight that merely
17

---

Published as a conference paper at COLM 2025
matching human-labeled error steps is insufficient for ensuring genuine understanding of mathematical errors.
Direct Critic
Critique of Studentt Solution Steps
Math Problem
denafie obber tiddng number hardrniee parza jes de
aflw
wnlly noss
any
Student Solution
deat tn sar.
Shpp1
oue anngart poicon deess to
criique 2tep2
Correct Conceptual
Thes dent' to nélonde the number dit tmapt The piopia & sic having
Tris yn cveltty bufff
enccly then bectume
exacly
unds lacter dut an (g)
pake
uade
Conclusion Critique
Though Matching Labelee
anteed by coogle pesphe irro tpmeee
Stop
wourly porteeeee inctiiti
Correction of Student Solution
(2) -115
cleer bun
Hevii corneced cightei wouch up shep there paseedr gart prople
Stop1
bost
in te cuant
lagusi, enogle
koveii aken ros fetow is in comvie den keme Che wiich rewerrrs lodking
peners venc's sach ween innelved aelest bendiinie
coealit
Error Stop Labelded by Human
Insfffcctvvn Wrong Anewer
Leads
Stop
Ground Truth Final Answer
The gosk fore groch whhar euch verto cigprs o' lrent Ede persin chales singled
Figure 10: Although the critic correctly identifies Step 2 as the error step (matching human annotation), it fails to understand the underlying mathematical concept of graph theory, leading to an incorrect correction of 22 handshakes instead of the true answer 12.
Duder Dinudee
Student Solution
Wioing Answer
Direct
Stope
Ineffectioon Crikke
newaaar (2))
with Cotlic Stop
Brror Step Lubeled by Human
Ground Truth Finel Answor
Figure 11: Despite matching the human-labeled error step (Step 4), the critic provides con- flicting feedback and fails to recognize the fundamental issue in applying the Pythagorean theorem with perpendicular medians, leading to an incorrect solution.
18

---

Published as a conference paper at COLM 2025
Direct
Student Solution
Droo Step Lobeted by Humon
Truth Ansower
seffectioo Wivinn Aroouce
The Brnor Strp Wronng thrugg Lotvetee
Figure 12:
The critic matches Step as problematic but misunderstands the key issue in finite geometric series calculation, resulting in an incorrect final value of 2047 2047/2048 /0448.
Therefore, we augment ProcessBench's protocol by requiring that models must not only identify the correct error step but also provide correction that leads to mathematically valid solution. This stricter requirement helps ensure that models demonstrate genuine understanding of the mathematical concepts and errors involved.
G.2 Allowing Step-Level Flexibility
Our second adaptation addresses an inherent ambiguity in error identification: in many cases, mathematical errors can reasonably be attributed to multiple consecutive steps. Through our analysis, we found numerous instances where the exact "error step" is de- batable, with both the preceding and following steps being valid points of identification. As shown in Figures 13 to 15, these cases illustrate how mathematical errors often span multiple steps, making strict step-level matching overly rigid for meaningful evaluation.
Direet Crile
Cnlique
Math Problem
ofiiga aap
Error Partially Located
Student
and Olerr Aaudar Eor andeche
dendes
rerpeniy Dirry Awnder
witme j
for Zuea divamane
Conslution Crilique
One Step
mon
Missed
Correction of Student Solution
Labeled Error
Step by
Ground Truth Final Answer
Accurate Correction & Answer
Figure 13: In this cherry-and-cheese danishes problem, while the human annotator labels Step 4 as the error, the true conceptual error begins in Step 5 where the student miscalculates the solution. The model still achieves correct final answer despite identifying different step.
19

---

Published as a conference paper at COLM 2025
Direct Cribi
Math Problem
Com comen pesteeeeenn volee
Student Solution
tep!
Clique Pagg
a sosca how disted une au uo 12
2ep2
chourd
Sepp
Step
whilk o
Mixoed With Labaled Error
Correction of Studentt Solution
Error Stop Labeled by Human
Ground Truth Final Answer
cnlting inoough
Correction Ansieer
Figure 14: In this probability problem, while the annotator marks Step 2 as the error, the fundamental misconception in Step (overcounting combinations) directly leads to the final incorrect probability.
Direct Critic
Steps
Poor Step Seginentation
Math Problem
Student Solution
we Srree lo and dedesss olee dideerteee
Lve by Foart, And weenne wne deaerity link
uet to decoccof
pour our
caat the ansedestto meaa astrustt an
jabter sentine guri for couker
Seroel
Conduution bon of Crtiique
dein of toual
Mixsed With Labeled Error
Student Solution
whe coteuaty sieeac the somete
Error Step Labeled by Human
Aoee alee www.triccccc....
Ground Truth Final Answer
Perdii nemer a Jevi
1/0 verlee)
Accurata Conrection &
Figure 15: In this remainder calculation problem, the error could be attributed to either Step 3 (pattern identification) or Step 4 (pattern application), as they form a continuous chain of incorrect reasoning.
To account for this ambiguity, we introduce a +1 step tolerance in matching model predic- tions with human annotations. This modification better reflects the reality of mathematical error analysis while still maintaining rigor in evaluation.
These adaptations result in a more meaningful evaluation protocol that better captures models' true understanding of mathematical errors and their ability to guide effective corrections.
20

---

Published as a conference paper at COLM 2025
Ditribution
0.28
0.28
0.28
Erose Peation
Figure 16: Distribution of first error positions identified by self-critic across different mathematical domains.
H Distribution of First Error Step identified by Self-Critic
I Classify Solutions into Correct and Incorrect
Again use 1eeen2.5-2B-Instruue itself to classify solutions into correct and incorrect ones. We present the system prompt in the following Figure 17:
you
Studion Solation.
and compare fhail answors.
Flrst, volidate the input:
Varily that Studint solutions are present and num-emply
Soudont solution
placeholder,
"ERROR: (Student) soution is missing imvolic
drvalidation passes, proceed with analysis:
Extract ninl Answers from Studont Solution:
ONLY look explicitty final
\bowedjansweri notation
condlssive
numerical reclls cieaely marked es
NOI intfe answers from work pariai culoulations
answer found, respond
"ERROR Cannot dentity explictt thai answer in Stude Solucion"
Combare
your response following
sorr hiral Avenoo Staket telatien
timele proidem leswer: Ceid at
bultipll parta. "oare (O) (sslag",, "Pars (B)) (valie",, etc.
Corclusion
Figure 17: System Prompt to classify solutions into correct and incorrect ones.
J Self-Training Details
Here we present the detailed configuration for self-training of Qwwen2.5-7BBInsstrutt We utilize open-instruct (Wang et al., 2023) for our continued supervised fine-tuning implemen- tation. The training was conducted on servers, each equipped with 8 NVIDIA A100 GPUs (32 GPUs in total), with total training time of several hours?.
The key haper-parameters for training are follows:
Batch size: 256
2The exact training time may vary depending on the specific hardware configuration and system load.
21

---

Published as a conference paper at COLM 2025
Learning rate: 5e-6
Number of training epochs: 1
Warmup ratio: 0.03
Model parallel size:
Total GPUs: 32 (4 servers 8 A100 GPUs)
For reproducibility, we gradient checkpointing and mixed-precision training (FP16) to optimize memory usage. The training was performed using DeepSpeed ZeRO-3 for efficient distributed training.
22


Reviews:
[{"review_id": "DTfJH7r3De", "reviewer": "Reviewer_whUs", "summary": "This paper presents SCRIT, a framework to improve the critique abilities of LLMs without relying on supervision from humans or stronger models. The authors' proposed method includes 1) a contrastive critic approach, where the model critiques a given solution by comparing it to a correct reference solution, and 2) a self-validation stage, which ensures that only critiques which lead to verifiably correct corrections are used for training. The authors evaluate their approach on mathematical reasoning, showing improvements on critic and correction tasks as well as error identification tasks, compared to the base model (Qwen2.5-72B-Instruct).", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The topic is relevant and the paper is well-written and motivated. Given the focus on mathematical reasoning, the evaluations are sufficient to show improvements compared to the base model, and illustrate the gap to SOTA performance.\n- The proposed method is straightforward and generates quality critique data without relying on human annotators or stronger models, making it more scalable and cost-effective.", "reasons_to_reject": "- The approach is math focused and depends on the availability of reference correct solutions, which limits generalization. It is not clear if the approach can be extended to other domains and more open-ended problems.\n- Some prior work in this domain are not referenced in the paper. For example:\n   - \"Shepherd: A Critic for Language Model Generation\" (Wang et al., 2023)\n   - \"CriticEval: Evaluating Large Language Model as Critic\" (Lan et al. 2024)", "questions_to_authors": "Questions:\n- In Tables 1 and 2, what do the bold numbers indicate? In some columns it is the highest number, whereas in others it is not.\n- In Table 1, what is your hypothesis for SCRIT giving more improvements on some datasets (e.g., MMLU) vs others? \n- In Table 1, under \"critic on balanced solutions\", SCRIT seems to hurt performance on 3 out of 8 datasets compared to the base model. Why is that?\n- Have you done any qualitative analysis of the generated critiques? Are there any systematic failure patterns observed in SCRIT generated critiques?\n- How does the choice of the base-model affect the outcome? Can SCRIT improve smaller models?\n\nPresentation:\n- The problem formulation is for mathematical problems (section 2), however, in the introduction, it is suggested that the approach is general but the authors decided to apply it to math as a testbed.", "extra_scores": {}}, {"review_id": "LbxFt54q1Y", "reviewer": "Reviewer_jYD3", "summary": "The study introduces SCRIT, a self-supervised framework that enhances LLMs' abilities to critique and correct solutions using only self-generated data. Experiments show that SCRIT improves error detection and correction accuracy without relying on human or stronger model supervision.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-motivated and easy to understand.\n2. The proposed SCRIT introduces a self-evolving critique mechanism that eliminates the need for human or stronger model supervision, addressing an important challenge in scalable oversight.\n3. SCRIT achieves consistent and significant improvements across multiple math benchmarks.", "reasons_to_reject": "1. Reliance on Reference Solutions:\nWhile SCRIT avoids the need for human supervision, it still relies on access to correct reference solutions. This may not be practical in many real-world scenarios where such reference solutions are unavailable or difficult to obtain.\n\n2. Limited Domain Evaluation:\nThe evaluation is restricted to mathematical reasoning tasks, where reference solutions are well-defined and readily accessible. It would be beneficial to validate the approach on tasks from other domains, particularly those where reference answers are ambiguous or not easily determined.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eWUwAgDoU0", "reviewer": "Reviewer_DGVo", "summary": "This submission proposes a self-evolving critic framework, which trains LLMs with self-generated data. The framework also employs a contrastive-critical approach and a self-validation scheme to improve the model quality. It applies the framework to Qwen2.5-72B-instruct model, and the resulting LLM shows consistent improvement on multiple tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A novel self-evolving critique framework that does not require external supervision\n2. Consistent performance gains over multiple tasks.", "reasons_to_reject": "1. Weak baselines: The submission only compares SCRIT with Qwen2.5-72B-instruct and o1-mini, while other strong recent methods such as CriticGPT, Reflexion are not compared.\n2. The experiments mainly focus on mathematic problems, while other reasoning tasks, e.g, logical reasoning, planning, and coding, are not considered, while may limit the generalisation ability of the proposed framework.", "questions_to_authors": "please see above.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents SCRIT, a framework to improve the critique abilities of LLMs without relying on supervision from humans or stronger models. The authors' proposed method includes 1) a contrastive critic approach, where the model critiques a given solution by comparing it to a correct reference solution, and 2) a self-validation stage, which ensures that only critiques which lead to verifiably correct corrections are used for training. The authors evaluate their approach on mathematical reasoning, showing improvements on critic and correction tasks as well as error identification tasks, compared to the base model (Qwen2.5-72B-Instruct).

Reasons To Accept:
* The topic is relevant and the paper is well-written and motivated. Given the focus on mathematical reasoning, the evaluations are sufficient to show improvements compared to the base model, and illustrate the gap to SOTA performance.
* The proposed method is straightforward and generates quality critique data without relying on human annotators or stronger models, making it more scalable and cost-effective.
* The proposed SCRIT introduces a self-evolving critique mechanism, addressing an important challenge in scalable oversight.
* SCRIT achieves consistent and significant improvements across multiple math benchmarks.

Reasons To Reject:
* Reliance on Reference Solutions: While SCRIT avoids the need for human supervision, it still relies on access to correct reference solutions. This may not be practical in many real-world scenarios where such reference solutions are unavailable or difficult to obtain.
* Limited Domain Evaluation: The evaluation is restricted to mathematical reasoning tasks, where reference solutions are well-defined and readily accessible. It would be beneficial to validate the approach on tasks from other domains, particularly those where reference answers are ambiguous or not easily determined.
Decision: Accept