Teaching Language Models to Critique via Reinforcement Learning
Authors: Zhihui Xie
Jie chen
Liyu Chen
Weichao Mao
Jingjing Xu
Lingpeng Kong
Venue: ICML 2025 poster (2025)
Keywords: LLMs
learning to critique
generative reward models
reinforcement learning
Primary area: deep_learning->large_language_models
TL;DR: We propose CTRL, a framework that trains LLMs to critique without human supervision, enabling them to supervise stronger models and achieve test-time scaling through iterative critique-revisions.

Abstract: Teaching large language models (LLMs) to critique and refine their outputs is crucial for building systems that can iteratively improve, yet it is fundamentally limited by the ability to provide *accurate judgments* and *actionable suggestions*. In this work, we study LLM critics for code generation and propose $\texttt{CTRL}$, a framework for $\texttt{C}$ritic $\texttt{T}$raining via $\texttt{R}$einforcement $\texttt{L}$earning, which trains a critic model to generate feedback that maximizes correction performance for a fixed generator model without human supervision. Our results demonstrate that critics trained with $\texttt{CTRL}$ significantly enhance pass rates and mitigate compounding errors across both base and stronger generator models.
Furthermore, we show that these critic models act as accurate generative reward models and enable test-time scaling through iterative critique-revision, achieving up to 106.1\% relative improvements across challenging code generation benchmarks.

Teaching Language Models to Critique via Reinforcement Learning
Zhihui Xie
Jie Chen '2
Liyu Chen
Weichao Mao? Jingjing Xu 2 Lingpeng Kong'
https://critio-rl.co wit..i.tt.b.ii. ic
Critic
Abstract
CTRL
Qwen2.5 Coder
GPT-4o
Teaching large language models (LLMs) to cri- tique and refine their outputs is crucial for build- ing systems that can iteratively improve, yet it is fundamentally limited by the ability to pro- vide accurate judgments and actionable sugges- tions. In this work, we study LLM critics for code generation and propose CTRL, a framework for Critic Training via Reinforcement Learning. which trains critic model to generate feedback that maximizes correction performance for a fixed generator model without human supervision. Our results demonstrate that critics trained with CTRL significantly enhance pass rates and mitigate com- pounding errors across both base and stronger generator models. Furthermore, we show that these critic models act as accurate generative re- ward models and enable test-time scaling through iterative critique-revision, achieving up 106. 1% relative improvements across challenging code generation benchmarks.
Swen2.5 Coder Generator
26
GPT 4o Generater
16
14
24
Pack@1
12
22
10
20
8
18
Critiquerrvissin Iteration
Critique-revision
Figure I. Perforrancce scaling our CTRL critic (finetuned on Qwen2.5- 2022555Coder3222 (ooder-3BBIIn Ins, henceforth Qwen2.5-Coder. compared to other critics across different generators on CodeContests. CTRL demonstrates strong critiquing capabilities not only when paired with its base model but also with a stronger generator (GPT-40, right). Shaded regions indicate standard error across 5 seeds.
degradation. To address this, existing approaches primarily rely on reward models (Sun et al., 2023; Yuan et al., 2024) or automated verification tools (Gou et al., 2023; Chen et al., 2023). However, these mechanisms often fail to provide actionable guidance reward models compress complex evaluation criteria into simplified numerical signals (Gao et al., 2023; Pan et al., 2024), while verification tools gener- ate lewelevel execution traces that do not directly translate to high-level fixes (Zhong et al., 2024). Even in domains like code generation (Li et al., 2022; Sun et al., 2024) where such feedback mechanisms are readily available, previous work (Zheng al., 2024) well our experiment (Table reveal that such feedback alone struggles drive meaning- ful improvements. At the heart of this issue lies the feedback bottleneck: feedback needs to both accurately discriminate the correctness solutions and provide informative yet actionable suggestions for improvement.
1. Introduction
Recent advances in Large Language Models (LLMs) have sparked interest in their potential for self-improvement through iterative feedback mechanisms (Pan et al., 2023). Methods like Reflexion (Shinn et al., 2024) and Self-Refine (Madaan et al., 2024) demonstrate that LLMs can, in prin- ciple, critique their own outputs and generate refined re- sponses. This self- self-improvement paradigm offers promis- ing direction toward more autonomous Al systems that can learn from their mistakes.
However, the effectiveness of such self-imprrvemment mech- anisms remains challenging in practice. Huang et (2023) demonstrate that without appropriate external feedback, such seff-improvement loops may lead to performance
To address these challenges, we propose CTRL (Critic Training via Reinforcement Learning). framework that decouples the critic model from the task lask-perorming model (e.g., GPT-4o) and focus on developing specialized critic that can effectively drive the lask-performing model to- ward optimal solution generation through iterative critique- revisions (Figure 2). This decomposition naturally intro- duces well-defined proxy task for training the critic model:
'Equal contribution 'The University of Hong Kong Bytedaacce Sced. Correspondence to: Zhihui Xie <zhxieml@ gmail.com. com>..
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Teaching Language Models to Critique via Reinforcement Learning
HUMAN
ASSISTANT
del resuthskrraysell queriac be]] Ii ko . Lie]jet:
plane
heap
You positive k.
20 which contains following querles:
results II
quariesl0 D. y): Buldd an abscacle at cocndiaaee is y0 the prane. is guarenteeed that there is no obsttacle at this
for y in cuer les: dietace abxodl abaly) heneo.hesppure
coondinate when this query is made.
distanne)
After each
find disance the anth
obstacte from the origin.
Resur integer ama results where resunsf]] denotes the kmth nearnss obstacto after query or resuntsf) if there
if lentnin hespt Ji ic rew.lis.spoond heap stow:
ane less than obstactes.
Note that instwaly there are no obstacles anyothene. The distacee of on obstacle coordinate Se, yt the origin givan by [4] lvl.
return resuits
CRI TIC
ASSISTANT
Anttetie
The browided solution antempts to use a min-heap to keep track of the distances of obstactes from the origi. However, it incorecctyy assumes that accessing ing iin heapph 1J' will give the kith amallest eement directy, not case heaps
did res.ttskraaybbef querle Liff adfiilll, kt listjoe)
smallest elemem wements index does not yleld correct 1-th smallest value.
heep
always root. Therefore, atterppting
Further more, the approsen does wifl clently handle the requitement to stways return the kith neareet obstacte wher each query, especially considoring the larg constrains where the number dueries can
112 times 10150.
for % ir quariesc
distence slolls) abalyy
Improeement sungessoons
eenmmax hespl 4:) heape, hwappushy max heap, odlatancel elf distar oo maak heap
To adie ves these issuee, cons iaw meintaining a belenooe data structure that el csse effideett now tion and retrecee of the l-th smallest clement. A balanced binory search tree or more advanced structure like a skip list could pocentally work. these might be complex implement
given constroims.
hevens heeepappmmeeeeeee heapor www.pushimepccccccmmmmm heep, adietincn
Alsematively, focus using a heap besed strategy more affectivvilyy Ore possible aspreach inoolves maintakiing max-heap of aize to store the smatiest distanoes encoumered so for When a new obstacle is added, ca cutste its distance and compare with the argest element the heap the root the max- heapi the new cistance is smaller, replace laress exement with the new cistance, ensures that the heap always containa the smallost distancess. distanes.
If ie entmas heap! W- it.
resulls append)-mee. (ep (51))
wose: resuits apperdd-1
Deseal anferment: Incornect
return resurts
Figure 2. Illustration of the critique- process for a coding problem. Top: An initial solution is proposed by the task lask-perfrmming using min-hcap approach. Bottom: The critic identifies flaws in the implementation (incorrect heap access and inefficient query handling) and suggests specific improvements, leading to a corrected max-heap solution. This example is taken from critiques of CTRL on LiveCodeBench, which demonstrates how structured feedback from the critic can guide meaningful improvements in code generation.
while directly evaluating the quality of generated critiques remains challenging, the effectiveness of critic can be measured by its ability to drive the task task-performing model toward correct outputs. Though such indirect optimization signals lead to a large space of possible critiques and there- fore high variance during training, we address this through a two-stage pipeline: first synthesizing high-quality critiques using execution feedback for supervised finetuning, then optimizing the critic through Group Relative Policy Opti- mization (GRPO; Shao et al. 2024).
critic significantly reduces the number of revision itera- tions needed, leading to both lower token consumption and higher success rates. Our empirical analysis (Figure 4) demonstrates that this efficiency stems from reduced error compounding- critic effectively identifies and corrects mistakes early, guiding the model toward more direct solu- tion paths without compromising solution quality.
Our work makes four key contributions: (I) We propose CTRL, a novel framework that decouples critic LLMs from task-performing models and trains them through two-stage GRPO to guide code improvement. (2) Through extensive evaluation on programming benchmarks, we demonstrate that CTRL significantly outperforms both self-critique meth- ods and approaches using stronger critic models. (3) We establish that relatively weaker critic models can effectively guide stronger task aas-performing models, demonstrating a promising weak-to-strong generalization phenomenon in LLM guidance. (4) We show that a trained critic enables test-time scaling through iterative critique-reviiions achiev- ing up to 106.1% % and 23.5% relative Pass I improvements on the challenging CodeContests benchmark when paired with its base model and a stronger model, respectively.
Through extensive evaluations on diverse benchmarks including CodeContests (Li et al., 2022), Live- CodeBench (Jain et al., 2024), MBPP+ (Liu et al., 2024a), and JudgeBench (Tan et al., 2024), we demonstrate that training with CTRL significantly outperforms both self-critique approaches and methods using stronger critic models. Notably, we observe remarkable generalization capabilities of the decoupled critic LLM across different problem domains and model scales. Our experiments demonstrate that relatively weaker critic models can effectively guide stronger task-performing models such as GPT-4o (Table 3). exhibiting similar phenomenon to weak-to-strong generalization (Christiano al.. 2018: Burss et al., 2023), where weaker models can be trained to effectively supervise more capable ones.
2. Preliminaries and Motivation
The success of iterative improvement methods critically depends on their ability to leverage feedback to improve
Furthermore, CTRL enables efficient test-time scaling (Fig- ure 1). By providing targeted and actionable feedback, our
2

---

Teaching Language Models to Critique via Reinforcement Learning
solutions. Formally, let a be input problem and y be candidate solution, with R(y) being the evaluation function that returns I if y is correct and 0 otherwise. Starting with an initial proposal distribution
Strong Criliquing
Weak Criliuuing
Critiquing
T[- æ), the iterative process generates subsequent solutions by incorporating feedback f( z, yı) and produce the next solution y/+1-
0.7
0.6
In this context, the effectiveness of such feedback mecha- nisms relies on two key capabilities: (I) discrimination the ability to evaluate and rank solutions, and (2) critiquing the ability to provide actionable feedback for improvement. While discrimination has been extensively studied (Gao et al., 2023), we focus on the critiquing ability and pro- pose to characterize it through the transition dynamics of a Markov chain (Meyn & Tweedie, 2012) governing the correctness of the iteratively refined solutions {R(y:)}::
0.4
0.3
Discrimination lty
Strong
Weak
0 10
20
30
40
S0
60
Number of Attempts
Figure 3. Simulation results showing success probability (Peorees) as function of the number of attempts, comparing different levels of critiquing and discrimination ability.
P(R(yo) = 1) = Pinit,
P(R(g++) = 1 R(ns) = 1) = Pen: P(R(U++) = R(y)) = 0) = Pow
abilities. While perfect abilities are not necessary, systemat- ically improving these capabilities particularly the ability to generate actionable critiques - is crucial for realizing the full potential of iterative refinement approaches.
where Pee represents the critiquing ability to avoid turning correct solutions into wrong ones, and Pew captures the helpfulness of the feedback in improving the solution.
3. Method
With analysis presented Section 2, our goal is teach LLMs the ability of critiquing without human supervision. We propose CTRL, a two-stage training approach: (1) syn- thesizing high-quality critiques by reasoning about execu- tion feedback, then (2) refining the critic through reinforce- ment learning. Once trained, the critic model can be used at test time, paired with any generator models, to iteratively refine solutions. A complete overview of the pipeline is pro- vided in Appendix A, with critique samples in Appendix E.
Varying Critiquing Ability. To understand the impor- tance of the critiquing ability, we conduct simulations across different levels of critiquing strength while lever- aging discrimination to aggregate the final solutions. We consider Pmnit = 0.1 and three scenarios: (1) No critiquing (Pew = Pcc), a special case representing methods that in- dependently sample from the base distribution, or equiv- alently best-of-n sampling (Sessa et al., 2024); (2) Weak critiquing (Pee 0.7, Pew 0.15); and (3) Strong cri- tiquing (Pee 0.9, Pew 0.3). For each scenario, we first generate solutions based on the specified transition dynamics, then apply the discrimination ability to select the best promising solution, and plot the final correctness probability against the number of attempts n. We present more details Appendix C.I.
3.1. Problem Statement
We focus on code generation as our primary domain as it provides clear objective metries through test cases, fol- lowing previous work (McAleese et al., 2024). Given programming problem æ (specified in natural language) and a solution y (code implementation), our goal is to enable iterative refinement of solutions, which centers on two key components: (1) a generator model T | xx) that proposes solutions, and (2) a critic model Co(elo, y) that provides textural feedback for improvement.
Observations & Takeaways. As shown in Figure 3, our analysis reveals several key findings: (1) Strong critiquing abilitics significantly improve success rates compared to no critiquing, with performance gains visible even with weak critiquing, aligning with recent empirical findings (Huang al., 2023). (2) Strong critiquing ability can compensate for weaker discrimination
Let D TT) be our training dataset, where each problem x; is paired with unit tests T, We have access sandbox environment that executes code against test cases, which serves as the evaluation func- tion R(y) that returns 1 if passes all tests, 0 otherwise. Notably, the sandbox does not assist critique generation at test time. While not required, we treat the generator model as a black-box, allowing our approach to build upon existing
Assumptions.
system with weak discrim- ination but strong critiquing feedback can outperform one with stronger discrimination but no critiquing ability. (3) The benefits of critiquing compound with more iterations, while approaches with no critiquing plateau quickly. These findings highlight that effective iterative improvement re- quires careful attention to both discrimination and critiquing
3

---

Teaching Language Models to Critique via Reinforcement Learning
strong generators without access their parameters.
Table I. Critique-revision performance (Pass 1, %) on CodeCon- tests. We fix the generator model to be Qwen2.5- Codee,, and compare zer-shot performance with critique performance using different feedback mechanisms. x represents conducting iterative critique-revision times. 'using unit tests for generation.
Objective. While directly measuring the helpfulness of critiques remain challenging, we can define proxy task that evaluates whether the critique leads to improved so- lutions. Given an initial solution y' ~ T(- | z), the critic analyzes it and produces textual feedback c. The generator then uses this feedback to revise the solution, producing an improved output y. Let (a, 3²) represent the problem- solution pair. Our objective is to train the critic model Co to maximize the expected solution quality:
Pass 1
A
A
Zero-shot
7.88
0.00
0.00
Execution Feedback (FF)
8.97
2.42
1.33
Self-critique w/ EF"
11.76
3,88
0.00
Self-critique
8,36
2.30
1.82
Critique w/ CTRLSFT
8,36
3.52
3.03
= [(0)==aaDxxxppoootta (R(y),,
(1)
Critique w/ CTRL
11.76
4.73
0.85
Critiquex 2 w/ CTRL
14.18
7.27
0.97
Critique 3 w/ CTRL
15.15
8.12
0.85
where (v = 2)m(y | c) denotes the improved solution distribution through marginalization over possible critiques. Notably, although Equation (1) defines single-turn critique-revision task, we observe that the trained model generalizes to multi-turn revisions (Section 4.2).
Table 2. Discrimination performance (F1 score, %) CodeCon- tests.
Passed
Failed
Macro
Qwen2.5- S-ddeer
88.21
34.16
61.19
Defining the Critique Space. We structure the critique space into three components (Figure 2): (1) an analysis of the solution's 's strengths and weaknesses, (2) actionable improvement suggestions, and (3) final judgment of cor- rectness (correctinncorreet) During inference, these compo- nents enable iterative critique- revision, where the process stops once the judgment indicates the solution correct. This design balances diserimination and critiquing, both essential for iterative refinement, as discussed in Section 2.
CTRLsFT
95.54
41.26
68.55
CTRL
93.19
45.02
69.10
Critique Synthesis. Building on the above insight, we de- velop a critique synthesis approach that leverages execution feedback to train models in generating effective critiques. Our approach samples high-quality synthesized critiques from a hinted distribution Ce(c | 2, h), where hints h are constructed by analyzing initial solutions y' through sand- box execution. We map different execution outcomes to specific hint templates as shown in Table 7: (1) for pass- ing solutions, we encourage concise positive feedback; (2) for completely failing solutions, we suggest restarting from scratch; and (3) for partially failing solutions, we provide the exact error message and test case details to help pinpoint the issue.
3.2. Stage Execution-guided Critique Synthesis
Although conceptually straightforward. learning effective critiques is challenging due the large critique space, where only small fraction leads to successful revisions. Our experiments with Qwen2.5-Coder (Hui et al., 2024) (Ta- ble 1) show that models struggle to generate informative critiques for self-improvement, aligning with previous find- ings (Huang et al., 2023). Self-critiuue without additional feedback yields minimal gains (7.88% -> 8.36%) and rarely converts incorrect solutions to correct ones, highlighting the limited ability of models to correct their own mistakes.
Supervised Finetuning. Similar to context distilla- tion (Snell et al., 2022; Guan et al., 2024), we exclude these hints and conduct supervised finetuning to encourage the model to internalize the critiquing strategies. We observe leveraging execution feedback for supervised finetuning beneficial mainly in two aspects: (1) helps learn the for- mat; (2) while it marginally improves the critique-revsiion performance due to the high frequency of instructing correct solutions to wrong (Table 1), it substantially boosts discrim- ination by providing ground-truth correctness (Table 2).
Reasoning over Execution. While the initial critiquing ability is limited, previous work (Ni et al., 2024) has shown that LLMs can effectively reason over execution feedback. Table 1 demonstrates that when LLMs reason over execution feedback to generate critiques (Self-critique w/ Execution Feedback), they achieve substantial improvements, as com- pared to directly using raw execution feedback for revisions (11.76% vs. 8.97%). This suggests that while directly using raw execution feedback is inefficient, we can leverage the model's ' reasoning ability over execution feedback to help generate more accurate and informative critiques.
3.3. Stage II: Reinforced Critique Generation
While our critique synthesis approach with predefined tem- plates provides a strong foundation, may not capture all nuanced feedback scenarios required for complex program- ming tasks. To overcome this limitation, we formulate
4

---

Teaching Language Models to Critique via Reinforcement Learning
critique generation as reinforcement learning problem, allowing the eritic to adaptively learn feedback strategies through direct optimization of solution improvement.
where clip, represents clipping the value to [1 E, +
Gatleia)) and D = Ga(ailz) log Ca(eil=) - denotes the KL regularization term that alleviates over-optimization.
Co(c/l)
Our goal is to maximize the performance in Equation (1). To optimize Co. one natural approach is using policy gradient methods (Sutton et al., 1999):
4. Experiments
We conduct extensive experiments evaluate our method's effectiivneess across multiple benchmarks. Our evaluation focuses on two key aspects: (1) the accuracy of the critic in identifying solution correctness, and (2) the quality im- provement achieved through critique-guided revisions.
V VoByrno [R(v)
Vo Colela)ntaleeo)))()))
=oo Σ R(v) Σ Coldz))(ylk,c))
Σ R(y) Σ VeColcl)mm(lk..
4.1. Setup
Training Data. We use TACO (Li et al., 2023), a dataset containing 26,443 programming problems collected from competitive programming platforms like CodeForces and LeetCode. Each problem includes a natural language de- scription and multiple test cases. Due to noise in the original dataset (malformed test cases and contaminated problems), filter the dataset to 18,820 problems for training, with details presented in Appendix C.3.
The double summation over both solution space y and feed- back space e introduces high variance in gradient estimates:
Var(Vo) [VV - [[V))> < |a|| [C]-
where [V] and [C] are the sizes of solution and critique spaces respectively. In this scenario, using value networks to predict credit assignment remains challenging, as we observe significant instability when using Proximal Policy Optimization (PPO; Schulman et al. 2017)- the learned networks produce noisy estimates of critique quality. We present detailed experimental observations in Appendix D.
Models. We base our critic model on the open-source Qwen2.5-Coder-Ins (Hui et al., 2024) model. During train- ing, we fix the generator model to be Qwen2.5-Coder-In itself. For evaluation, we assess the trained critic's perfor- mance by pairing with various generator models for initial solution generation and subsequent revision, comparing against other LLM critics such GPT-4o.
Variance Reduction. To combat these variance issues, we adopt Group Relative Policy Optimization (GRPO; Shao et al. 2024) that avoids using value networks for learn- ing credit assignment and reduces variance through group- based relative advantages. Specifically, for each problem- solution pair 2 = 8'), we sample a group of critiques fci, c2....,ca! from (-1+ and compute advantages:
Benchmarks. We evaluate our approach on three program- ming benchmarks and one general-domain benchmark: (I) CodeContests (Li et al., 2022), a collection of challenging competitive programming problems; (2) LiveCodeBench (24.08-24.11) (Jain et al., 2024), a curated set of recent programming challenges designed to minimize data con- tamination; (3) MBPP+ (Liu et al., 2024a), extension of the MBPP benchmark (Austin al., 2021) focused on fundamental programming tasks; and (4) JudgeBench (Tan et al., 2024), where we evaluate the model's effectiveness as a generative reward model for comparing solution pairs.
R(y:)= PG OG
A;
where yi cz) is the improved solution generated using critique c, and μa and aG are the mean and stan- dard deviation of rewards within the group. This approach normalizes rewards across different problem types and nat- urally focuses training on problems where critique quality can make a meaningful difference, as problems that are too easy or too hard produce zero relative advantages. The final training objective is:
Metries. To evaluate critiquing ability, we use three met- ries: Pass@ measures the success rate of the final solutions, A represents the fraction of initially incorrect solutions that become correct after revision, and A represents the frac- tion of initially correct solutions that become incorrect after revision. For discrimination ability, we employ Fl score when evaluating single solutions, and accuracy when com- paring paired solutions in Judgebench, as the latter involves binary decisions between two alternatives.
J(0)= = Eg Ee Co
1
G
Co(c,l2)
Co(cilz) Caaa (c:|2) ).i)
min
AG, clip,(
(
G
Co (cilz)
1=1
BDx. (Col (Ce)(Cret)
5

---

Teaching Language Models to Critique via Reinforcement Learning
Table Performance comparison across different generators and benchmarks. We evaluate different configurations, with critique-revision representing an iterative process where critic model provides feedback to guide solution improvement. Pass@) @ shows the success rate, while A and A, indicate percentage wrong solutions being correctly revised correct solutions being revised to wrong solutions, respectively. Results are averaged over 5 random seeds.
CodeContests
LiveCodeBench
MBPP+
Average Pass @
Pass
D
A
Pass@1
A
A
Pass@1
A+
Qwen2.5 2- C-Coder Generator
Zero-shot
7.88
0.00
0.00
30.54
0.00
0.00
77.83
0.00
0.00
38.75
Single-turn Critique-revision
Critique w/ Qwen2.5-Coder
8.36
2.30
1.82
32.14
2.50
0.89
77.83
3.49
3.49
39.45
Critique w/ GPT-4o
10.67
4.85
2.06
32.32
2.32
0.54
77.46
3.81
4.18
40.15
Critique w/ CTRL
11.76
4.73
0.85
33.21
3.39
0.71
78.84
2.43
1.43
41.27
Multi-turn Critique-revision
Critiquex5 5 w/ Qwen2.5-Coder
9.21
3.76
2.42
29.64
2,14
3.04
76.03
3,81
5.61
38.30
Critiquex5 w/ GPT-4o
12.48
7.03
2.42
32.86
4.82
2.50
74.60
4.34
7.57
39.98
Critiquex5 x5 w/ CTRL
16.24
9.21
0.85
33.39
3.75
0.89
78.68
3.23
2.38
42.77
GPT-4o
Generator 32.32
Zero-shot
20.61
0.00
0.00
0.00
0.00
77.67
0.00
0.00
43.53
Single-turn Critique-revision
Critique w/ Qwen2.5-Coder
20.24
3.52
3.88
35.36
3,93
0.89
76.67
0.85
1.85
44.09
Critique w/ GPT-4o
20.97
2.30
1.94
34.82
2.68
0.18
77.41
1.01
1.27
44.40
Critique w/ CTRL
23.03
4.97
2.55
33.39
2.14
1.07
77.83
0.53
0.37
44.75
Multi-turn Critique-revision
Critiquex5 Qwen2.5-Coder
19.52
5,21
6.30
35.54
5.36
2.14
76.08
1.53
3.12
43.71
Critiquex5 5 w/ GPT-4o
20.61
3.39
3.39
35.18
3.21
0.36
76.61
2.06
3.12
44.13
Critiquex5 w/ CTRL
25.45
7.88
3.03
34.11
3.21
1.43
77.94
0.79
0.53
45.83
Execution Sandbox. We employ SandboxFusion (Liu al., 2024b) execution environment, which provides unified interface for evaluating solutions across training data and benchmarks through both function-biig based and stan- dard input-output formats.
Pass@ rate from 11.76% 15.15% while maintaining low regression rate A, of 0.85%. This demonstrates that our critic provides consistently reliable feedback across multiple revision iterations, unlike baseline approaches that accumulate errors, as discussed below.
4.2. Evaluating Critics for Iterative Critique-revisions
CTRL Mitigates Compounding Errors. Figure 4 further illustrates this stability advantage while both Qwen2.5- Coder and GPT-4o show increasing error compounding rates over iterations, CTRL maintains significantly lower rate, enabling reliable multi-round improvements.
To evaluate the effectiveness of CTRL, we present compre- hensive analysis of critique-revision strategies with different feedback mechanisms on CodeContests in Table 1. The discrimination performance of critics is shown in Table 2, while results across different benchmarks and generators are presented in Table 3.
CTRL Generalizes to Different Generators and Tasks. While we train the critic model with Qwen2.5-Coder the generator, as shown in Table 3, our approach generalizes well across different programming tasks. Notably, a weak critic model trained against itself can assist stronger model (GPT-4o), providing evidence for scalable oversight (Chris- tiano al., 2018; Kenton al., 2024).
RI Significantly Boosts Critiquing Ability. Table shows that our RL-trained critic significantly outperforms baseline approaches, achieving 11.76% pass@) I rate com- pared to 7.88% with zero-shot generation. This substantial improvement builds upon much reduced regression rate A than its SFT counterpart (0.85% vs. 3.03%).
Performance Scaling with Probiem Difficulty. As shown in Figure 5, critique-revision approach demon- strates increasingly substantial relative gains as both itera- tion and problem difficulty increases, revealing that CTRL is particularly effective for complex tasks, where iterative refinement through targeted critique and revision yields the most significant benefits compared zero-shot generation.
CTRL Enables Test-time Scaling. As shown in Table 1, our approach enables test-time scaling through iterative critique-revisions. Notably, despite training exclusively on single-turn critiquing tasks, CTRL generalizes to multi-turn settings. By increasing the number of iterations from one to three (Critique x3 w/ CTRL), we further improve the
6

---

Teaching Language Models to Critique via Reinforcement Learning
--- CTRL
Qwen2.5-Codd Codeer
Critic
GPT-40
Coding
Overall
85-7
80
70A
@wen2.5-Cole Generator
T-do Genérator
75.4
Accuracy (96)
Regression d.
2.5
5
6
70
64366-3
643613
65.7
2.0
52,452.3
3
50
1.0
2
0.5
40
Liama 3. 1-108--a
GBM Gewma-2am
GP7-49
Cooeer 320-4a
Claude- 3 S-nnnee
Ous
of V-mini
#t al-previtew
0.0
Qwenz Coseen 5
Critiquerrviiiion Iteration
Critique-revision Iteration
Figure 6. Model performance comparison on JudgeBench.
Figure Compounding emror analysis. Regression rate measures the frequency of correct initial solutions being revised into incor- rect ones. Shaded regions indicate standard error over seeds.
-specic evaluations. This suggests that our CTRL enables effective discrimination capabilities that generalize beyond the training domain.
4.3. Evaluating Crities as Generative Reward Models
One advantage of unifying textural feedback is to bal- ance discrimination and critiquing abilities. To assess our critics' discrimination capabilities, we evaluate them on JudgeBench (Tan al., 2024), a comprehensive benchmark containing 350 GPT-4o completions across categories span- ning general knowledge, reasoning, mathematics, and cod- ing. This setup presents a challenging
4.4. Analysis
To better understand how CTRL boosts iterative refinemen,, we further conduct analyses on the similarity between orig- inal and revised solutions, execution time changes, and critique characteristics. Our findings reveal several key pat- terns in how different critique methods influence the process of critique-revision.
test in two aspects: (1) our critics must evaluate outputs from a more capable model than their training distribution, and (2) they need to generalize to broader domains beyond coding tasks. This evaluation scenario is particularly inter- esting as examines whether relatively weaker models can be effectively trained to judge outputs from more powerful models.
The Effect of Generator Ability. As preliminary analy- sis before finetuning experiments, examine how model sizes affect critique-revision performance using Qwen2.5- Coder-Ins models (7B, 14B, and 32B) in an inference-only setting, comparing zero-shot generation against critique- revision with critiques generated by another critic model conditioned on execution feedback. Table reveals that critic capability significantly influences improvement poten- tial while smaller critics (7B) often lead to performance degradation, larger critics (32B) consistently yield better out- comes, achieving up to 50% improvement when paired with
As shown in Figure 6, CTRL critic achieves competitive performance compared to stronger models such as Claude- 3.5-Sonnet. Notably, while our critic is specifically trained on programming tasks, it maintains comparable overall ac- curacy (64.3% while demonstrating superior performance
Critique-revision Iteration
Table 4. Relative improvement (%) CodeContests when com- paring critique-revision (using crities conditioned on execution feedback) against zero-shot generation, different generator- critic size combinations. Results are from inference-only experi- ments hefore any finetuning.
25
Pass Rate (%))
20
15
10
Critic 14B
Generator
Avg.
7B
32B
7B
-33.33
22.22
-11.11
-7.41
Easy
Medium
Hard
14B
-9.09
-9.09
9,09
-3.03
32B
0.00
30.00
50.00
26.67
Figure Comparison of pass@1 I rates by problem difficulty with CTRL crities on CodeContests. Results are averaged over 5 seeds.
Avg.
-14.14
14.38
15.99

---

Teaching Language Models to Critique via Reinforcement Learning
Distribution of Solution Similaribies
Similarity Score Comparison
Table 5. Timeout rate and Pass@1 (%) on LiveCodeRench. While CTRL approach achieves higher pass rates, it tends to generate more comprehensive solutions that take longer to execute.
1,00
Smillerit Stote
2.5
0.75
Denalty
2.0
CTRL Mears 0.313
0.50
0.5
0.25
Timcout Rate (L)
Pass@ (t)
0.0
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
1.2
0,00
Selt-ortique CTRL
Zero-shot
10.54
30.54
Similarity Score
Critique w/ GPT-4o
8,93
32.32
Critique w/ CTRL
16.61
33.21
Figure 7. Comparison of solution similarities between original and revised code guided by CTRI on CodeContests. Left: Distribution of similarity scores for self-critique and our CTRL, method. Right: Box plot showing the statistical distribution of similarity scores. Lower scores indicate more substantial revisions.
Huang et al. (2023) demonstrates that without appropri- ate external feedback, such self-improvement loops may lead to performance degradation. Our work addresses these challenges by learning specialized models that can provide effective feedback for improvement.
similarly-sized generators. The results also highlight the im- portance of critic-generater size relationships, as critics less capable than their generators typically degrade performance. These findings motivate us to focus our subsequent finetun- ing experiments with CTRL on 32B models to maximize the benefits of critique-revision.
LLM Critics. Several approaches have been proposed to train LLMs as critics for various purposes, including gen- erative reward models (Ankner et al., 2024; Xiong et al., 2024) and scalable oversight (Saunders et al., 2022; Ken- ton et al., 2024). These approaches either learn from hu- man feedback (Wang et al., 2023; McAleese et al., 2024) or much more capable models' outputs (Xi et al., 2024), with recent work exploring reinforcement learning to im- prove feedback generation (Akyürek al., 2023; Yao et al., 2023). Our approach differs in three key aspects: (1) lever- aging execution feedback and model reasoning to synthesize high-quality critiques, (2) introducing variance reduction techniques to stabilize training, and (3) requiring only single- round critiuue-rviisin interactions. Additional discussion on related work is provided in Appendix B.
CTRL Prevents Similar Revisions. We analyze how dif- ferent critique methods influence solution revisions by mea- suring code similarity scores between original and revised solutions, as described in Appendix C.4. As shown in Fig- ure 7, self-critique tends to make conservative modifications with higher similarity scores (mean 0.482), while our CTRL method proposes more substantial changes (mean 0.313). This suggests CTRL is more willing to recommend major structural revisions when needed, rather than just local opti- mizations, which may explain its superior performance in improving solution quality.
Scaling Test-Time Compute. Recent work has explored various upproaches to improve model performance at test time without fine-tuning (Snell et al., 2024). While exist- ing approaches focus on techniques like repeated sampling with proper selection mechanisms (Brown et al., 2024) and more sophisticated modular frameworks with existing mod- els SSacoFFalcon et al., 2024), we instead investigate test- time scaling through a decoupled critic model trained to provides targeted feedback to guide solution improvements. Notably, while Saad-Falcos et al. (2024) demonstrates that strong models can serve as effective critics, their approach struggles with code generation tasks.
CTRL Trade-offs between Accuracy and Efficiency. While our critique approach improves solution ac- curacy on LiveCodeBench, we observe a notable increase in timeout rates. Solutions guided by CTRL exhibit a timcout rate of 16.61%, higher than both zero-shot (10.54%) and GPT-4o critic (8.93%). However, even with more timeouts, CTRL still achieves better overall Pass @ accuracy. This suggests that our approach tends to generate more compre- hensive solutions - while these may take longer to execute, the solution quality is guaranteed.
5. Related Work
6. Conclusion
sell-emproeeenttt of LLMs. Recent work has explored various approaches for LLMs improve their outputs tonomously, including self-reflection (Shinn et al., 2023; Feng et al., 2024), self-critique (Madaan et al., 2024: Shinn et al., 2024), debates (Irving et al., 2018; Michael et al., 2023; Khan et al., 2024), and training models to self- correct (Welleck et al., 2022; Kumar et al., 2024). However,
We present CTRL, a reinforcement learning framework for training critie LLMs to provide effective feedback for itera- tive refinement. Our trained critic demonstrates significant improvements over baselines across multiple benchmarks and enables efficient test-time scaling through iterative critique-revision. - notably, even when guiding stronger generators. While this work focuses on improving pass
8

---

Teaching Language Models to Critique via Reinforcement Learning
rates, future directions include optimizing for efficiency and safety, and extending training pipeline towards multi- turn critique revision. We hope this work inspires further research into scalable LLM improvemnt through rein- forcement learning.
Cui, G., Yuan, L.., Ding. N., Yao, G., Zhu, W. Ni, Y., Xie, G., Liu, Z., and Sun, M. Uttrafeedbcck: Boosting language models with high-quality feedback, 2023.
Feng, X., Wan, Z.,, Fu, H., Liu, B., Yang, M., Koushik, G. A., Hu, Z., Wen, Y., and Wang, Natural language reinforcement learning. arXiv preprint arXiv:2411.1425/ 2024.
Impact Statement
This work aims to advance the field of Machine Learning by introducing framework for training LLM critics. While this research has the potential to improve the reliability and robustness of AI systems, we have not identified any immediate societal concerns requiring specific attention. However, as with any Al technology, careful consideration should be given to its broader deployment and potential misuse.
Gao, I.., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Gou, Z., Shao, Z.,, Gong, Y., Shen, Y. Yang, Y., Duan, N., and Chen, W. Critic: Large language models can self- correct with tool-interactive critiquing. arXiv preprint arXiv:2305. 11738, 2023.
Guan, M. Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Heylar, A., Dias, R., Vallone, A., Ren, H., Wei, J., al. Deliberative alignment: Reasoning enables safer lan- guage models, arXiv preprint arXiv:2412.16332 2024.
References
Akyürek, A. F,, Akyürek, E., Madaan, A., Kalyan, A., Clark, P.. Wijaya, D., and Tandon, N. R14f: Generating natural language feedback with reinforcement learning for re- pairing model outputs. arXiv preprint arXiv:2305.08844 2023.
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., and Zhou, D. Large language mod- els cannot self-correct reasoning yet. arXiv preprint arXhv:2310.01998 2023.
Ankner, Z., Paul, M., Cui, B., Chang, J. D., and Am- manabrolu, Critique-out-lon reward models. arXiv preprint arXiv:2408.1 11791, 2024.
Hui, B., Yang. J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409, 122224499.1166.. 12186, 2024.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M.. Le, Q.. al. Program synthesis with large language models. arXiv preprint arXiv:2108.0773: 07732, 2021.
Irving, G., Christiano, P., and Amodei, D. Ai safety via debate. arXiv preprint arXiv:1805.00880 00899, 2018.
Jain, N., Han, K., Gu, A., Li, W.-D,, Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free eval- uation of large language models for code. arXiv preprint arXiv:2403.0720 077974, 2024.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324334) 1952.
Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.217881 2024.
Kenton, Z., Siegel, N. Y., Kramár, J., Brown-Cohen, J., Al- banie, S., Bulian, J., Agarwal, R., Lindner, D., Tang, Y., Goodman, N. D., et al. On scalable oversight with weak Ilms judging strong 1lms. arXiv preprint arXhv:2407.04622 2024.
Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet. A., Joglekar, M., Leike, J., et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09339. 2023.
Khan, A., Hughes, J., Valentine, D., Ruis, L., Sachan, K., Radhakrishnan, A., Grefenstette, E., Bowman, S. R., Rocktäschel, T., and Perez, E. Debating with more persua- sive 1lms leads to more truthful answers. arXiv preprint arXiv:2402.0677 2024.
Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05118, 2023.
Kumar, A., Zhuang, V., Agarwal, R., Su, Y., Co-Reyes, J. D., Singh, A., Baumli, K., Iqbal, S., Bishop, C., Roclofs, R., et al. Training language models to self-correct via reinforcement learning, arXiv preprint 2rX.v:2409.12917, 2024.
Christiano, P., Shlegeris, B., and Amodei, D. Supervis- ing strong learners by amplifying weak experts. arXiv preprint /rXiv:18/00 08555, 2018.

---

Teaching Language Models to Critique via Reinforcement Learning
Li, R., Fu, J., Zhang, B.-W., Huang, T., Sun, Z., Lyu, C., Liu, G., Jin, Z., and Li, G. Taco: Topies in algorithmic code generation dataset. arXiv preprint arXiv:2312.14452, 2023.
Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L.., Ward, J., and Leike, Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.0588 2022.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Fecles, T., Keeling, J., Gimeno, F., Dal Lago, A., al. Competition-Iovel code generation with alpha- code. Science,
Klimov, O. Proximal policy optimization algorithms.
arXiv preprini (00Xvv:1707.064444 2017.
Sessa, P. G., Dadashi, R., Hussenot, L., Ferret, J., Vieil- lard, N., Ramé, A., Shariari, B., Perrin, S., Friesen, A., Cideron, G., et al. Bond: Aligning Ilms with best-of-n distillation. arXiv preprint arXiv:2407. 2eii::240..1622,,, 2024.
2022.
Liu, J., Xia, C. S., Wang, Y., and Zhang. L. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024a.
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song. J., Bi. X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Liu, S., Zhu, H., Liu, J., Xin, S., Li, A., Long, R., Chen, L., Yang. J., Xia, J., Peng, Z., et al. Fullstack bench: Evaluating Iims as full stack coder. arXiv preprint DoXiv:24/2.00555 2024b.
Sheng, G., Zhang, C., Ye, Z., Wu, X., Zhang. W., Zhang, R., Peng, Y., Lin, H., and Wu, C. Hybridflow: A flexible and efficient rIhf framework. arXiv preprint urXiv:2409. 19256, 2024.
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye. S., Yang, Y., et al. Self-refine: Iterative refinement with self- feedback. Advances in Neural Information Processing Systems, 36, 2024.
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal rein- forcement learning. Advances in Neural Information Processing Systems, 36:8634-8652, 2023.
McAleese, N., Pokorny, M., Uribe, J. F. C., Nitishinskaya, E., Trebacz, M., and Leike, J. Llm crities help catch Ilm bugs. arXiv preprint arXiv:2407.00215. 2024.
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal rein- forcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Meyn, P. and Tweedie, R. L. Markov chains and stochas- tic stability. Springer Science & Business Media, 2012.
Snell, C., Klein, D., and Zhong, R. Learning by distilling context. arXiv preprint arXiv:2209. 15199, 2022.
Michael, J., Mahdi, S., Rein, D., Petty, J., Dirani, J., Pad- makumar, V., and Bowman, S. R. Debate helps super- vise unreliable experts. arXiv preprint 231::2311.08702 2023.
Snell, C., Lee, J., Xu, K., and Kumar, Scaling 11m test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint arXiv:2408.033: 043114 2024.
Ni, A., Allamanis, M., Cohan, A., Deng, Y., Shi, K., Sut- ton, C., and Yin, P. Next: Teaching large language models to reason about code execution. arXiv preprint arXiv:2404. 14662, 2024.
Sun, Q., Chen, Z., Xu, F., Cheng, K., Ma, C., Yin, Z., Wang, J., Han, C., Zhu, R., Yuan, S., et al. A survey of neural code intelligence: Paradigms, advances and beyond. arXiv preprint axXiv:2403.1473 2024.
Pan, J., He, H., Bowman, S. R., and Feng, S. Spontaneous reward hacking in iterative self-refinement. arXiv preprint vxXiv:2407.04542 2024.
Sun, Z., Shen, Y., Zhang, H. Zhou, Q. Chen, Z., Cox, D. D., Yang. Y., and Gan, C. Salmon: Self-alignment with principle-following reward models. CoRR, 2023.
Pan, L., Saxon, M., Xu, W., Nathani, D., Wang, X., and Wang, W. Y. Automatically correcting large lan- guage models: Surveying the landscape of diverse self- correction strategies. arXiv preprint WxXv:2308..03188 2023.
Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Saad-Falcon J., Lafuente, A. G., Natarajan, S., Maru, N., Todorov, H., Guha, E., Buchanan, E. K., Chen, M., Guha, N., Ré, C., et al. Archon: An architecture search framework for inference- time techniques. arXiv preprint arXiv:2409. 15544, 2024.
Tan, S., Zhuang, S., Montgomery, K., Tang, W. Y., Cuadron, A., Wang, C., Popa, R. A., and Stoica, I. Judgebench: A benchmark for evaluating llm-based judges. arXiv preprint arXiv:2410. 12.04,274.. 2024.
10

---

Teaching Language Models to Critique via Reinforcement Learning
Wang, T., Yu, P., Tan, X. E., O'Brien, S., Pasunuru, R., Dwivedi-Yis Yu, J., Golovneva, O., Zettlemoyer, I.., Fazel-Zarandi, M., and Celikyilmaz, A. Shepherd: A critic for language model generation. arXiv preprint arXiv:2308.04553 2023.
Welleck, S., Lu, X., West, P., Brahman, F., Shen, T., Khashabi, D., and Choi, Y. Generating sequences by learning to self-correct. arXiv preprint 2.0i::2211.00055.. 2022.
Xi, Z., Yang, D., Huang, J., Tang, J., Li, G., Ding, Y., He, W., Hong, B., Do, S., Zhan, W., et al. Enhancing Ilm reasoning via critique models with test-time and training- time supervision. arXiv preprint arXiv:2411.1633
2024.
Xiong, T., Wang, X., Guo, D., Ye, Q. Fan, H., Gu, Q.. Huang, H., and Li, C. Llava-critie: Learning evaluate multimodal models. arXiv preprint 12.07:2410.02712, 2024.
Yao, W., Heinecke, S., Niebles, J. C., Liu, Z., Feng, Y., Xue, L., Murthy, R., Chen, Z., Zhang, J., Arpit, D., et al. Retro- former: Retrospective large language agents with policy gradient optimization. arXiv preprint Sttv:2308.0215.. 2023.
Ye, Z.., Greenlee-Scott, F., Bartolo, M., Blunsom, P., Cam- pos, J. A., and Gallé, M. Improving reward models with synthetic critiques. arXiv preprint arXiv:2405.20850 2024.
Yu, Y., Chen, 7., Zhang, A., Tan, L., Zhu, C., Pang. R. Y., Qian, Y., Wang, X., Gururangan, S., Zhang, C., al. Self- generated critiques boost reward modeling for language models. arXiv preprint 2rXiv:2411.16666 2024.
Yuan, W., Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J. Self-rewarding language models. arXiv preprint 1:Xi::2401.10022 2024.
Zheng, K., Decugis, J., Gehring, Cohen, T., Negrevergne, B., and Synnaeve, G. What makes large language models reason in (multi-turn) code generation? arXiv preprint 10000:2410.08105. 2024.
Zhong, L., Wang. Z., and Shang, Ldb: A large language model debugger via verifying runtime execution step-by- step. arXiv preprint 4xXiv:2402.16900 2024.

---

Teaching Language Models to Critique via Reinforcement Learning
A. Pipeline
As shown in Figure 8, our pipeline consists of two main training stages. (1) The SFT training stage first generates initial solutions that are validated through execution feedback, followed by critique generation where the generator learns to provide critiques based on execution feedback. These components are then used train the final critic model through supervised finetuning. (2) The RL training stage leverages the critic's feedback to guide the generator in producing improved solutions, which are validated in sandbox environment.
DATA & ROLES
SFT TRAINING 1: Initial Solution Generation
Context Distillation
Prompt
Unit
Prompt
Denerator
Solution
Sandboxx
Feedback
Step 2: Critique Generatign
LLM
Generator
Prompt
Solution
Feadback
Exac
Generator
Crilique
Sandbox
Critic
Step 3: Supervised Finetuning
Prompt
Solution
Craiquee
Criticc
RL TRAINING
Prompt
Solution
Critic
Critique
Rl. Trainer
Generator
New Solution
Sandibox
Pass?
Figure 8. Overview of our two-stage training pipeline CTRL.
B. Supplementary Discussion of Related Work
Table 6 categorizes prior methods into reward models, generative reward models, and critic models. Reward models like Standard RM (Bradley & Terry, 1952) and SynRM (Ye et al., 2024) focus on discrimination by outputting scalar rewards r but lack refinement or critique supervision. Generative reward models, such as CLoud (Ankner et al., 2024) and Critic-RM (Yu et al., 2024), enhance discrimination by producing both rewards and critiques c, but their critiques primarily serve by-product for rewards rather than actionable refinement suggestions. Critic models, including UltraCM (Cui al., 2023), Shepherd (Wang et al., 2023), and CriticGPT (McAleese al., 2024), focus on generating critiques but rely heavily on human annotated critique data, which limits scalability. In contrast, CTRL unifies discrimination and refinement by generating actionable critiques without direct supervision, leveraging execution feedback and reinforcement learning to enable scalable, iterative improvement.
Table 6. Comparison of reward models, generative reward models, and critic models.
Methods
Input
Output
Discrimination Refinement
Critique Supervision
Standard RM (Bradley & Terry, 1952)
y
SynRM (Ye et al., 2024)
y,
UltraCM (Cui et al., 2023)
!/
Shepherd (Wang et al., 2023)
I, y
C
CriticGPT (McAleese et al., 2024)
T, y
C
CLoud (Ankner et al., 2024)
r
X
Critic-RM (Yu et al., 2024)
V
X
CTRL (Ours)
x, y
C. Implementation Details
C.1. Simulation
In our simulation (Section 2), we model the iterative refinement process using a Markov chain with parameters Pinit Pee. and Pew to represent the initial correctness, the probability of maintaining correctness, and the probability of turning incorrect
12

---

Teaching Language Models to Critique via Reinforcement Learning
solutions correct, respectively. Critiquing ability is controlled by varying Pee and Pew (e.g., strong critiquing: Pee 0.9, Pew 0.3; weak critiquing: Pez 0.7, Pew 0.15), while discrimination ability is adjusted via true positive rate (TPR) and false positive rate (FPR) (e.g., strong discrimination: TPR 0.7, FPR 0.2; weak discrimination: TPR 0.6, FPR = 0.3). For each setting, we simulate n refinement steps using Python, generating solutions based on refinement probabilities, applying a classifier to predict correctness, and selecting the best solution from predicted correct ones. The results are computed over 50,000 iterations and plotted to analyze the impact of critiquing and discrimination on final success rates. Specifically, the two processes processes-only -only using discrimination and using both discrimination and critiquing - are illustrated in Figure provide clearer understanding of our simulation setup.
1/1
1/2
T
y
x
yib
CO
1/
yn
V3
Un
w/o Critiquing
w/ Critiquing
Figure Graphical models for refinement processes: (left) only using discrimination (best-of-n sampling) and (right) using both discrimination and critiquing (sequential critique-revision).
C.2. Prompt Templates
The generator model <t 2)) is implemented as a simple zero-shot generation process, where the model generates solution y directly from the problem statement without additional context feedback. The critic model Co(c| x, y), as described in the main paper, generates textual feedback using a structured prompt that incorporates the problem x; the solution y, and explicit instructions to provide actionable and formatted suggestions. The improved solution distribution ( xr ,), 3) implemented as a two-turn process: the first turn, the generator model drafts the initial solution y' conditioned on the problem x as the user message; in the second turn, the critique is presented as the user message, and the model revises the solution, conditioned on x, y'. and
Critique-revision.
Execution-guided Critique Generation. To generate high-quality critiques (Section 3.2), we leverage execution feedback from a sandbox environment that evaluates the initial solution y' against the test cases T for the problem æ. The execution results are mapped to predefined hint templates, which guide the critique generation process. The critic model prompted with structured template incorporating the problem z, the solution y', and the corresponding hint enabling to produce actionable and context-aware feedback. To prevent hallucination, critiques that explicitly reference the hints are filtered out. This ensures that the generated critiques grounded in observable failures while effectively supporting solution refinemett.
C.3. Training
Data Curation. Our data curation process starts with the TACO dataset (Li et al., 2023) and handles both function-based and iuput-output-based programming problems. We filter out malformed problems by removing those containing image tags and unusual HTML spans. For unit tests, we process them differently based on their type: function--asee tests are converted to assertion statements, while onput-output tests are standardized into a sandbox format with stdin-stdout pairs. We exclude problematic unit tests such as those with malformed string inputs (containing assignments or unexpected list operations) or invalid function calls. To avoid contamination, we further exclude 47 problems that overlap with our evaluation benchmarks. The final dataset deduplicated based on problem descriptions, resulting in 18,820 problems.
Supervised Finetuning. We leverage the synthesized critiques to perform supervised finetuning (SFT) on the model, enabling it to generate improved solutions. For each problem, we sample one initial solution and one corresponding synthesized critique, and train the model on these problem-sole -olutioo--oiiiiiuee pairs. The training process follows the
13

---

Teaching Language Models to Critique via Reinforcement Learning
Prompt Template for Critique Generation
are tasked and lyzing answer to problem providing conatruct feedback, providd + direct aonutions.
Problem description:
<problem>
(problem)
</problem>
Answer:
<answer>
[anawer]
</answer>
Structure response using the follewing format (without <format> taga)
<tormat>
Anslysie:
(:nnalysin))
Improvement suggestions:
(:Suggestions]]
Overall Judgment: ([Dozreet/Tnoorreel]) </format>
Prompt Template for Execution-guided Critique Generation
You. are tasked with anelyzing an answer to problem and providing constructive feedback. NOT provide direct solutions.
Please corafully reason about the hint to guide the user. *mpoortant: NOT mention "the hint! your feedback.**
description:
<problem>
(problem)
</problem>
Anewes:
<answer>
(solution)
</answer>
<hint>
[hint]
</hint>
structure response using the following format (without <format> tags)
<format>
Analysis:
(:Analysiat)
improvement suggestions:
([Saggastians)]
Overall judgment: ([Correct/Intocree)) </format>
hyperparameters outlined in Table 8.
RL Training. We use VeRL (Sheng et al., 2024) as the codebase to optimize the model's generation quality. During RL training, we sample initial solutions for each problem and train the critic model on all corresponding problem-solution pairs. This approach helps mitigate overfitting by exposing the critic to a diverse set of solutions for each problem. The RL training process follows the hyperparameters outlined in Table
14

---

Teaching Language Models to Critique via Reinforcement Learning
Table Mapping between execution results and hint templates used for critique synthesis.
Execution Result
Hint
Success (100%)
The draft solution is correct. A concise and positive feedhack recommended. The draft solution is entirely wrong.
Failure (0%)
A concise feedback requesting a fresh restart recommended. Input: (input)
Expected Output: {expected outputt
Partial Success
Actual Output: _octull
The code block: "python crodebblock,
Runtime Error
raised {error).
Table RL Hyperparameteess
Table 8. SFT Hyperparameters.
Parameter
Value
Parameter
Value
Training Batch Size
1,024
Mini-Batch Size
256
Learning Rale
x 10
Group Size
8
Learning Rate Schedule
Cosine
Learning Rate
I 10
Training Batch Size
256
Coefficient
0.001
Maximum Token Length
2,048
Maximum Prompt Length
1,536
Training Epochs
Maximum Response Length
768
Mixed Precision Format
6
Temperature
1.0
Training Epochs
2
C.4. Evaluation.
During inference, we use a temperature of 0.7 for generating both initial solutions and critiques, while revised solutions are generated using greedy decoding. The maximum number of tokens generated is set to 1,024 for all stages.
Inference.
Reward Calculation. To calculate rewards for our JudgeBench evaluation (Section 4.3), we use critic model assess the quality of solutions. Specifically, we generate multiple critiques for each solution and aggregate the results through majority voting. For each solution pair, the critic model compares the frequency of being labeled as "Correct" to determine which solution is better. As shown in Figure 10(a), we find that the accuracy of this majority voting strategy improves as the number votes increases.
Code Similarity Calculation. To measure code similarity while accounting for semantically equivalent code with different variable names, follow (Zheng al., 2024) and implement two-step comparison approach. We first normalize the code by parsing it into an Abstract Syntax Tree (AST), systematically renaming variables to canonical forms, and converting back to consistently formatted text. We then compute a similarity ratio using Python's 's difflib SequenceMat cher, which represents the proportion of matching characters in the optimal alignment of the two normalized code sequences. This approach yields a score between 0 and 1, allowing us to identify structurally similar solutions regardless of variable naming choices.
15

---

Teaching Language Models to Critique via Reinforcement Learning
64.0
64.3
60
Accuracy (%)
50
50.0
40
30
23
m mmn
20
0.2
21
23
2
27
Number of Votes
-0.2
30
150
180
Step
(a) The effect of the number of votes the accuracy of majority voting in reward calculation. As the number of votes increases, the accuracy improves significantly, demonstrating the scalabil-
(h) Training curve of the value network in PPO, showing the predicted value training steps.
ity and robustness of the majority voting approach.
Figure (a) Majority voting accuracy improvement with increasing votes. (b) Training curve the value network.
D. The Credit Assignment Issue for Critic Training
Our initial attempts to train critics using Proximal Policy Optimization (Schulman et al., 2017) for RL training revealed challenges in credit assignment for critique generation, as evidenced by the unstable value predictions shown in Figure 10(b). This difficulty arises because the quality of a critique is inherently holistic - it depends on multiple interdependent aspects like accuracy, completeness, and constructiveness- teweenesssmakinnn making it hard for the value network to learn which parts of the generated critique contributed the final reward. These observations motivated our switch to GRPO, which cireumvents the value prediction problem by using Monte Carlo sampling directly estimate advantages, leading more reliable credit assignment without the need for potentially unstable value network.
E. CTRL Samples
Tables 10, 12 and 14, we present sample critiques generated by CTRL for solutions provided by Qwen2.5-Coder.
16

---

Teaching Language Models to Critique via Reinforcement Learning
Table 10. A sample problem from CodeContests (Li et al., 2022), with by the solution generated by Qwen2.5-Coder and the critique
provided by the CTRL critic.
Problem & Solution
Critique
had a positive integers a_1, a_2, ...
aczanged in cirele. For each pair of neighboring numbers (a_1 a_2, a_2 a_3, and a_ll,
i_(n 1 and R_D, and wrote downs are the numbers in the pair equal not.
Unfortunetely, you'vee
piece paper array Koreover, you are afraid that even information about equality neighboring elements may be inconsistent 8o, wondering: is there any array a which consistent with informatien
knalyais:
The provided solution attempts to construct an arrey based the sequence of "g' and 'w' but does not correctly handle the circular nature of the array and constraints imposed by these choracters. spacifically, the approach does not ensure consistency acrosa the entire ciroular arzangement, particulerly at the boundary where the last element connects back to the first.
are
have about equality or equaliylllyy of cortesponding pairs
can__frmm farn_form_ae (s)
The code assigns velues to the array elemeats based whether
len
current character 'g' 'N', but it does not account for potertial conflicta that arlse when the array wraps around. This the first and last elements might fail due to sroonaistanccess introducad earlier in the array construction process.
Ree will tzy to assiga values to the array a
Start with assigning to the first element
[i]
that the finsi check between
Tzy to assign values based on the given
conditions
for i in range (n):
slil 'g': alli 1) 8 a[i]
Improvement auggestions:
1. Coasider using a more systematic approach to of the array. possible methed is to usa a graph traverssl algorithn, treating as node and each 'gr or "g" as an edge with label indiosting equality or inequality.
else: alli
handle the circular
n a[i]
position
array
Check if
consisteat with sla if s[-1] rg' and a[0] 1- a[-1l: "ND"
2, Use unior-find data structure equivaleace classes of indices that must hold the value due '2' connections.
manage
if a[-1] =n eHe and a(0) == a(11] "MO"
3. Iterate through the string and apply the rules:
indicea if (8", ensure they are in differeat sets.
set;
return
4, After proceasing the string, verify that no conflicts exist in the union find structure, particularly focusing on the circuler connection between the first and last elements.
def main():
import eys
input syys.stdin.read. ht lines
int (input (0])
5. Tf na conflicta are found, it la possible to construct array that satlefies conditions; otherwise, it is not.
results []
range(L, input(i] rosults. for.rrearrppcctooooooooraarreeee
Overall judgment: Incorrect
for zesult in sesults: print (result)
mamo ()in()
moin
17

---

Teaching Language Models to Critique via Reinforcement Learning
Table 12. A sample problem from LiveCodeBench (Jain et al., 2024), with by the solution generated by Qwen2.5-Coder and the critique provided by the CTRI. critic.
Problem & Solution
Critique
ace given binary string A binary string satisfies the k-constraint if either following conditions helds:
inLeger
Analysis:
The provided solution atteapts
problem counting substrings based on the maximum occucrence rgr '1' up to 'k', it does not correctly account for all possible substrings that satisfy the k-constraint, The approach incorrectly caleulates
The number o'' in the string is at most k. The number of L's in the string is at
Retura inzeger denoting number substrings of that satisfy the k-censtraint.
intersection of subs:sings that satisfy both conditions without properly considering overlap batween these substrings. Additional the logic for counting substringe is flawed as it does consider every possible substring of 's' instead, it focuses on contiguous sequences of 'D's and '1'5 up 'k'.
class Solutions
def coun.KKonttraaitttuuttttti (eelf, 8: str, &t int) int: def count max_ohar, left 0 count current_count
Improvement auggastions:
resul: 0
1. **Re-evaluate the Approschrs: Instead of focusing on igguus sequences r0's and "1's, the solution should consider every possible aubstring of 's' and check if it satiafies the k-conatraint.
for right in ranga(1an s[right] max_char: curseat_count 1-
2. Ileraaee Over All Substrings Use nested loops to iterata over
while current_couus k: if s :left charchar: current_coure laft +=
possible substrings of ", For each subatring, count the oecurrences of ro' and '1' and check if either count is less 'k',
than equal
3. AsCount Valid Substringswx: Maintoin a counter to of many sunstrings constraiat criteria.
result 1- right left
return result
** ify the logiote: Consider using Python'a slicing capabilities to easily extract and eveluate each substring without unneressary checks for contigueus charscters.
count cout.abberrigg.wtttt....... '0', k) count_ie coutt_eubbtinngg..wit...... '1' k)
Overall judgment: Incorrect
return count_0s count_ls
18

---

Teaching Language Models to Critique via Reinforcement Learning
Table 14, A sample problem from MBPP+ (Liu et al., 2024a), with by the solution generated by Qwen2.5-Codor and the critique provided by the CTRL critic.
Problem & Solution
Critique
Analysis:
Write python function to check whether every odd iadex contains odd numbers of a given 1ist.
provided solution checks
parity ledd/aven nature) of the numbers at aed indi cea matches the index itself, This means it verifies that number
def add Lengeee_numaarre (an())))
odd index is cod, and numbee an even index is even. However, this does not correctly addrass requires checking whether every add index contains on number, not just matching the parity of the index.
res -
problem at hand, which
Iterate through each element 1n the array for i in range (n):
calculate the number of subarrays in which arr[i] is present
Improvement auggestions:
//
Review the logic usad in the function. Instead of number matches the parity of the index, the function should direatly is edd,
checking if parizy
count is odd, element the resul!
nimber at each
count $
arr [1.
Consider using more straight Forward loop or list comprehension achieve the desired result.
Overall judgment:
Incorrect
19


Reviews:
[{"review_id": "IhGw6i5RGl", "reviewer": "Reviewer_25hb", "summary": "Teaching large language models (LLMs) to critique and refine their outputs is crucial for building systems that can iteratively improve, yet it\nis fundamentally limited by the ability to provide accurate judgments and actionable suggestions. In this work, the authors study LLM critics for code generation and propose CTRL, a framework for Critic Training via Reinforcement Learning, which trains a critic model to generate feedback that maximizes correction performance for a fixed generator model without human supervision. Their results demonstrate that critics trained with CTRL significantly enhance pass rates and mitigate compounding errors across both base and stronger generator models.\n\n# Update after rebuttal\nThis paper is novel and clear in presentation, and the rebuttal resolves my concerns. I will keep my positive rating as my final score after rebuttal phase.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive and well-thought-out review! We appreciate your recognition of our work's novelty, effectiveness, experimental design, clear motivation, and solid supplementary material. Your positive feedback is truly encouraging.\n\nBelow, we address your concerns about computational costs and system complexity:\n\n**Cost for majority voting in JudgeBench evaluation**:\n\nThe base critic used for majority voting is our CTRL-trained Qwen2.5-Coder-32B model, the one used throughout our evaluation. The majority voting approach is primarily used for our JudgeBench evaluation (Section 4.3) to calculate pairwise accuracy.\n\nWhile generating multiple critiques may seem costly, modern inference engines significantly reduce this overhead through prefix caching (i.e., the same prompt is prefilled for generating multiple critiques). This majority voting approach is also used in recent work on generative verifiers [Zhang et al., 2024], which observe similar results to ours that majority voting boosts accuracy significantly.\n\nOur experiments show that performance plateaus after approximately $2^6$ critiques, with additional samples providing diminishing returns (64.3 at Maj@$2^7$ vs 64.0 at Maj@$2^6$). For cost comparison with Claude 3.5 Sonnet (which performs similarly on JudgeBench), we calculate per-problem inference costs using [Together AI pricing](https://www.together.ai/pricing) for Qwen2.5-Coder-32B and [Anthropic pricing](https://www.anthropic.com/pricing#anthropic-api) for Claude 3.5 Sonnet:\n\nCTRL with Maj@$2^6$:\n- Average prompt length: 2,067.17 tokens\n- Average response length per critique: 231.72 tokens\n- Input/output cost: $0.80/1M tokens\n- **Cost**: (2067.17 + 231.72 * 64) * 2 * 0.8 / 1M = $0.027 per problem\n\nClaude 3.5 Sonnet:\n- Average prompt length: 1,627.74 tokens\n- Average response length: 652.97 tokens\n- Input cost: $3/1M tokens\n- Output cost: $15/1M tokens\n- **Cost**: 1627.74 * 2 * 3 / 1M + 652.97 * 2 * 15 / 1M = $0.029 per problem\n\nNote that the factor of 2 accounts for calculation on both responses in CTRL and removing positional bias for Claude 3.5 Sonnet.\n\nThis analysis demonstrates that our method achieves comparable performance to state-of-the-art proprietary models at a similar or slightly lower cost.\n\n**Cost for fine-tuning**:\n\nWe acknowledge that RL training is computationally intensive, especially for large models. To manage this, we implemented several optimizations:\n- SFT stage: Our SFT on self-critique with execution feedback provides a strong initialization, significantly reducing the need for extensive RL exploration.\n- Optimization techniques: We leveraged fully sharded data parallelism (FSDP), gradient checkpointing, and sequence packing to reduce memory requirements and speed up training. At each step the critic model and the generator model are offloaded to CPU to save memory, allowing us to train 32B models with a minimum of 8 x 80G GPUs.\n\nAs open-source RL training frameworks [1,2] continue to evolve, we expect these costs to decrease further in the future.\n\n[1] Sheng G, Zhang C, Ye Z, et al. Hybridflow: A flexible and efficient rlhf framework[J]. arXiv preprint arXiv:2409.19256, 2024.\n\n[2] von Werra L, Belkada Y, Tunstall L, et al. TRL: Transformer Reinforcement Learning[CP/OL]. GitHub repository. GitHub, 2020. https://github.com/huggingface/trl.\n\n**Clarification on \"multiple critics\"**:\n\nWe want to clarify that our approach uses only a single critic model during training, not multiple critics as may have been implied. We sample multiple critiques from this single model to estimate advantages in GRPO.\n\n**Hyperparameter tuning**:\n\nFor hyperparameter selection, we relied on prior work in RL training that identified the most sensitive parameters for training stability: learning rate, KL coefficient, and group size for GRPO. Due to computation constraints, we performed a very limited grid search over these key parameters (< 10 runs), with final parameter selection based on performance on a TACO test subset. We report the final hyperparameters in Table 9.\n\nWe agree that more extensive hyperparameter search could potentially improve results. We will improve the final version of our paper with a more detailed description of our grid search process.\n\n**Sandbox outputs as correctness signal**:\n\nRegarding using sandbox outputs as correctness signals, we would like to clarify two contexts of “reward calculation”:\n- During CTRL training: We do use sandbox outputs (pass/fail) as the reward signal to train our critic model, as described in Section 3.1.\n- For JudgeBench evaluation: High-quality unit tests are not available for many tasks, especially general-domain questions. This practical limitation is precisely why CTRL is valuable - it internalizes sandbox signals during training, allowing critic models to generalize to new domains where explicit test cases are not available.\n\nThank you again for your insightful review and positive assessment of our work.", "claims_and_evidence": "I summarize the claims issued in the paper, which includes:\n- Challenge: Without appropriate external feedback, such self-improvement loops may lead to performance degradation.\n- Solution 1: Reward models: reward models compress complex evaluation criteria into simplified numerical signals\n- Solution 2: Automated verification tools: generate low-level execution traces that do not directly translate to high-level fixes\n- Important: Feedback needs to both accurately discriminate the correctness of solutions and provide informative yet actionable suggestions for improvement.\n\nAll the claims align with my previous knowledge and have clear evidence in related works.", "methods_and_evaluation_criteria": "Yes, the critique qualtiy is important for underlying task performing model. Through (1 supervised fine-tuning to format the answer and enhandle the discrimination ability with hint, and (2 further RL training the critique through GRPO with advantage normalized reward that is calculated by majority voting of additional critique fixed evaluation models, they propose CTRL framework which is Qwen-2.5 based critique that surpasses multiple competitors like GPT-4o. And the experiment results and additional analysis about compounding errors, Test-time Scaling and Evaluating Critics as Generative Reward Models demonstrate the effectiveness of approach.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design is abundant and comprehensive. But I have a major concern about the cost:\n1. For reward evaluation, the authors use majority of vote for each step. Specifically, they generate multiple critiques for each solution and aggregate the results through majority voting. What is the base critic and the count is from 2 to $2^7$, does that cost too much?\n\n2. We know that fine-tuning Qwen2.5-32B is costly with SFT and even more costly with RL, how the authors have such computing resource to afford fully fine-tuning with knowledge distillation from self-critique with execution feedback or RL training to optimize the GRPO goal with multiple critics?", "relation_to_broader_scientific_literature": "This work relates to LLM critics for code generation, which applies GRPO to reduce variance through advantage and ensembles and chain of thoughts based prompt engineering with external ground-truth assistance (execution feedback) to improve the code generation quality. We note that such critique model can also serve as generative reward model for other domain tasks in JudgeBench.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength:\nThe idea is novel and useful to improve the performance with solid experimental demonstration and analysis disucssion.\nMotivation is clear, every step has a clear purpose.\n\nWeakness:\n- The implementation of methodology and baseline comparison requires large computing resources and long-time RL training with sensitive hyper-parameter tuning, how the authors overcome the difficulty and conduct hyper-parameter search for such a complex system with many components is interesting to know.", "other_comments_or_suggestions": "The reward calculate is too costly, could the authors just use the sandbox outputs as the correctness signal to save time?", "questions_for_authors": "Please see weakness part in \"Other Strengths And Weaknesses\" section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FzBpIhtIlO", "reviewer": "Reviewer_MYma", "summary": "The paper presents a method to teach an LLM to critique the response of another LLM, specifically in the domain of coding contest problems. The problem is formalized as maximizing the probability of the latter LLM to succeed at providing a correct response after seeing a critique produced by the former, which is an RL problem. \n\nThe method consists of an initial SFT stage followed by an RL stage. In the SFT stage, the critique model is trained on filtered critiques produced using a method that includes execution of the code produced by the answer model. In the RL stage the critique model is trained using GRPO to optimize the success of the answer model. \n\nThe results show a significant boost in performance on CodeContests, and MBPP, and to a lesser extent on LiveCodeBench. The performance further increases with multiple rounds of critique and revision (even though this was not done during training).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and valuable feedback! We appreciate your recognition that our main claim `is well supported by the experiments` with `a number of interesting analyses` and that the paper `very clearly motivates its approach by citing relevant related work.` We address your primary concerns below:\n\n**Proof: Variance of the gradient**:\n\nWe provide a short proof in our [anonymous link](https://anonymous.4open.science/r/icml25-rebuttal/anonymous.pdf) that justifies our claim about gradient variance scaling with $|\\mathcal{Y}| \\cdot |\\mathcal{C}|$.\nThe key insights from this proof are:\n- When analyzing the variance of our policy gradient estimator, we find that under mild assumptions it scales proportionally with the product of answer and critique space sizes ($|\\mathcal{Y}| \\cdot |\\mathcal{C}|$).\n- In cases where covariance terms are significant, the scaling becomes even worse than linear, potentially leading to a super-linear increase in the overall variance.\n\n\nThis theoretical analysis provides a mathematical foundation for why standard policy gradient approaches struggle with critique-revision optimization, directly motivating our CTRL design choices that address this variance challenge. We will incorporate the proof in our final version.\n\n\n\n**Dataset overlap between TACO and CodeContests**:\n\nThank you for raising this question! We agree that while we discuss this issue in the paper and propose a decontamination method to mitigate this, it deserves more attention. We would like to clarify a few points:\n\n- **Actual overlap**: While the TACO paper mentions a significant overlap with CodeContests overall, the critical distinction is that we evaluate on CodeContests test set (165 problems), while the reported overlap primarily exists between TACO and CodeContests full set. We excluded the 47 problems we identified as direct overlaps between the TACO train set and the CodeContests test set.\n\n- **Performance on out-of-distribution benchmarks**: We would like to highlight that our method demonstrates significant improvement over the zero-shot baseline on LiveCodeBench with Qwen2.5-Coder as the generator model (30.54% -> 33.21% in Pass@1), outperforming GPT-4o and Qwen2.5-Coder as critic models. This demonstrates that our method does learn useful critiquing abilities that transfer beyond its training distribution.\n\n- **Additional experiment**: To further validate CTRL's generalizability, we have conducted **additional experiments** on IFEval [1], a benchmark for instruction-following capabilities. The results are presented in the anonymous link, Table 1. Despite not being trained on instruction-following data, CTRL boosts accuracy by over 2% for single-turn critique-revision with Qwen2.5-Coder as the generator, outperforming all the baselines including GPT-4o.\n\nWe agree that scaling up the data size and diversity is a promising direction to further improve the results. and consider this an important direction for future work, which we will highlight in the final version.\n\n[1] Zhou J, Lu T, Mishra S, et al. Instruction-following evaluation for large language models[J]. arXiv preprint arXiv:2311.07911, 2023.\n\n\n**Typos and suggestions**:\nThank you for your careful review! We appreciate your suggestion about the naming convention, and will rename $Q$ and fix the spotted typos in the final version.\n\nThank you again for your thorough review. Your feedback has helped us strengthen our work and identify important areas for improvement.", "claims_and_evidence": "The main claim of the paper, that one can train a model to critique using RL (and SFT), is well supported by the experiments. The motivating ideas and claims made in section 2 and 3 are very well supported by the cited related works.", "methods_and_evaluation_criteria": "The method is rather straightforward and sensible. Similarly the evaluation datasets and benchmarks are standard (though see comments on experimental design below)", "theoretical_claims": "The claim on page 5 that variance of the gradient scales with the size of the answer and critique space are not proven. Other than that the paper does not make significant theoretical claims.", "experimental_designs_or_analyses": "The paper trains on the TACO dataset and evaluates on CodeContests, but these contain many of the same problems. The appendix states that 47 problems were excluded, but the TACO paper explicitly mentions that about half the TACO problems are also in CC, so I consider this a no-go. Moreover, the results in Table 3 show that although CTRL performs well on CodeContests and MBPP+ (similar contamination risk), on LiveCodeBench it fares worse than using either GPT-4o or Qwen2.5-Coder as critique models. This suggests that the model has not learned generalizable critiquing strategies that work beyond the narrow (and contaminated) distribution of TACO/CC.\n\nAlthough this is a serious issue that must be discussed in more detail in the paper and acknowledged as a weakness, it is still interesting to see that the model succeeds in-domain. It is quite likely that drastically scaling up the data size and diversity could enable learning of generalizing critiquing policies.", "relation_to_broader_scientific_literature": "The paper very clearly motivates its approach by citing relevant related work. This is a very nice feature of the paper.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "The experimental section of the paper contains a number of interesting analyses.", "other_comments_or_suggestions": "- I would not call the critic model Q, since this has a well-established meaning in RL\n- typo on page 5, \"we samples\", \"and computes\"\n- The equation for J on page 5 shows Q(y|x) but I think it should be Q(c | x, y)?", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E8NUd4iCd8", "reviewer": "Reviewer_VcEc", "summary": "This paper presents the CTRL framework - a two-stage training approach that separates the critique function of a language model from its generative capabilities. The authors first synthesize high-quality critiques using execution feedback (running code trying to go through unit tests), which are then used in a supervised fine-tuning phase of a dedicated critique model. In the second phase, they use Group Relative Policy Optimization (GRPO) to further refine the critics through reinforcement learning so that the generated critiques directly help the fixed generator model to improve its output (measured in terms of test pass rates). Experiments on multiple coding benchmarks (e.g., CodeContests, LiveCodeBench, MBPP+) and JudgeBench have shown dramatic improvements in the performance of the framework, with reported relative improvements of up to 106% on some metrics. The framework not only improves the pass rate of the base generator, but also generalizes well when applying the critic to stronger models (e.g., GPT-4). In summary, this paper presents a novel, mathematically-based, empirically validated method for guiding LMs to provide actionable critiques that significantly improve the correctness of generated code.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive and thoughtful review! We appreciate your recognition of our work's `solid mathematical and theoretical foundation` and `adequate experimental validation.` We would like to address your concerns in detail:\n\n**Theoretical analysis**:\n\nOur work is primarily empirical. While we use mathematical frameworks to motivate our approach, our main contributions are demonstrating a practical framework for critic training and empirically validating its effectiveness.\n\nTo strengthen our theoretical analyses, we discuss more in detail:\n\n- Markov chain analysis: We provide theoretical analyses of Markov chain convergence properties in the [anonymous link](https://anonymous.4open.science/r/icml25-rebuttal/anonymous.pdf) complementing Figure 3's empirical findings.\n- GRPO properties: We show that GRPO achieves variance reduction by a factor of $G$, explaining its stability with large critique spaces. Our analysis also demonstrates that GRPO ensures monotonic improvement and converges to a local optimum under standard assumptions.\n- Weak-to-strong generalization: Our findings that weaker critics can guide stronger generators align with recent work (Burns et al., 2023; Kenton et al., 2024), though theoretical conditions for when this works remain an open research question.\n\n**CTRL does reach the highest level of innovation**:\n\nThank you for recognizing our `novel, mathematically-based, empirically validated method.` Our primary contributions include:\n\n- Problem formulation: We identify and formalize the critique generation problem through a Markov chain lens, distinguishing between discrimination and critiquing abilities.\n- System design: We propose a decoupled critic-generator architecture and a two-stage training pipeline specifically designed for critique learning.\n- Robust training: We demonstrate GRPO's suitability for critique training due to reduced variance in policy gradients, and demonstrate its effectiveness empirically.\n- Empirical findings: We provide extensive empirical evidence demonstrating that our framework enables test-time scaling, mitigates compounding errors, and facilitates weak-to-strong generalization.\n\n**Applying CTRL to domains where feedback is less binary**:\n\nFor domains where feedback is less binary, CTRL can leverage alternative reward mechanisms:\n\n- Learned reward models: Training a reward model from pairwise preference data (as in RLHF approaches) for tasks like safety or preference alignment.\n- Reference-based evaluation: For tasks with reference responses (e.g., translation, summarization), we can use metrics like log-likelihood on reference outputs or automated evaluation metrics (e.g., ROUGE and BLEU) as reward signals.\n\nCTRL is agnostic to the specific reward mechanism as long as it can distinguish between successful and unsuccessful revisions. This flexibility extends CTRL beyond domains with clear verification tools.\n\nNotably, we observe positive generalization results on IFEval, a benchmark outside our training distribution, as shown in our anonymous link.\n\n**Sensitivity to GRPO parameters**:\n\nWe relied on prior work in RL training that identified the most sensitive parameters for training stability. Due to computation constraints, we performed a limited grid search (<10 runs), with final parameter selection based on performance on a TACO test subset. We found our method relatively robust to GRPO hyperparameters within reasonable ranges:\n\n- Group size: Larger sizes improve performance (we used 8) but increase computation time\n- KL coefficient: While unnecessary, KL improves stability and KL=0.001 balances exploration and stability  in our experiments.\n- Learning rate: Most sensitive parameter; we found 1e-5 to be optimal via the grid search.\n\n**Qualitative assessment of critiques**:\n\nTo better assess the quality of the generated critiques,  we manually evaluated critiques on 50 randomly sampled critiques from CTRL against Qwen2.5-Coder's solutions on CodeContests. Our analysis revealed CTRL employs diverse debugging strategies: algorithm improvement (43 instances), static analysis (38), strategic debugging (11), and dynamic testing (10).\n\nThis distribution reveals that CTRL primarily focuses on structural code improvements and algorithmic enhancements rather than superficial issues. The prevalence of static analysis and refactoring strategies also suggests CTRL learns debugging approaches that align with human reasoning patterns.\n\n\n**Robustness to noisy feedback**:\n\nWhile we didn't explicitly test noisy feedback scenarios, our low regression rates ($\\Delta_\\downarrow$) demonstrate CTRL can distinguish correct implementations from problematic code, avoiding misleading critiques that worsen solutions. This inherently suggests robustness to noise, though we agree more systematic evaluation of noise tolerance is an important direction for future work.\n\nThank you again for your detailed review. We hope our response addresses your concerns and questions.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The paper makes reasonable empirical claims about critic training but lacks formal theoretical proofs for its Markov chain analysis and optimization properties. The most significant theoretical gaps are:\n\nMissing justification for Markov assumptions in iterative refinement\nUnquantified variance reduction claims for GRPO\nUnexamined optimization landscape characteristics\nThese limitations don't invalidate the experimental results but suggest opportunities for deeper theoretical analysis in future work. The community would benefit from formal proofs of:\n\nMarkov chain convergence properties\nGRPO variance bounds\nConditions enabling weak critic supervision of strong generators", "experimental_designs_or_analyses": "Yes, no further issues", "relation_to_broader_scientific_literature": "The paper's key contributions extend prior work on LLM self-improvement by addressing critical limitations identified in existing literature. While earlier methods like self-critique (Madaan et al., 2024) showed theoretical promise for iterative refinement, Huang et al. (2023) demonstrated their practical limitations due to compounding errors: a challenge CTRL directly addresses through its specialized critic training. The framework builds on weak-to-strong generalization concepts (Christiano et al., 2018) but extends them to cross-model critique-revision dynamics, showing smaller critics can effectively guide larger generators. This advances beyond traditional reward modeling approaches (Gao et al., 2023) that used scalar feedback, instead aligning with emerging work on generative reward models (Yu et al., 2024) while eliminating their reliance on human annotations. The test-time scaling mechanism through iterative revisions responds to Snell et al.'s (2024) call for compute-efficient inference methods, but introduces novel critique-driven iteration rather than simple sampling. By formalizing the critique process through Markov chain analysis, the work provides theoretical grounding to empirical observations in code generation studies (Zheng et al., 2024) about the importance of actionable feedback, while offering a generalizable framework that could extend beyond programming domains.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "## Strengths\n- Solid mathematical and theoretical foundation:\nThe iterative improvement process is described through Markov chain modeling, which clearly illustrates the impact of criticism and discrimination ability on the final success rate.\nNormalization of the strategy gradient using GRPO significantly reduces the gradient variance, thus making the reinforcement learning process stable and efficient.\n- Experimental validation is adequate:\nSignificant improvement is demonstrated on several code generation benchmarks (CodeContests, LiveCodeBench, MBPP+), and the critic's discriminative ability is also verified on JudgeBench.\nExperiments comparing self-criticism, raw generation, and improvements under different generation models show that CTRL effectively reduces error accumulation and dramatically improves pass rates\n\n## Weaknesses\n\n- Methods are not innovative enough:\nAlthough CTRL has done some work in combining executive feedback, supervised fine-tuning and reinforcement learning, it has overall borrowed existing ideas and techniques, such as self-criticism, RLHF, and GRPO methods.\nThe core idea of the method is mainly to combine existing techniques, and it does not propose a completely new mechanism in terms of theory or algorithm, so it may not reach the highest level of innovation.\n- The theoretical analysis is not deep enough:\nAlthough relevant RL objectives and GRPO updating strategies are given, there is a lack of theoretical proof of convergence and optimality of the RL part, and the approach relies more on experimental performance.", "other_comments_or_suggestions": "- Authors should discuss the possibility of applying CTRL to domains where feedback is less binary or difficult to obtain (e.g., essay writing, dialogue generation).\nFuture work could explore how to build alternative feedback mechanisms in these domains.\n\n- It would be useful to discuss the sensitivity of the method to various hyperparameters (e.g., group size in GRPO, strength of KL regularization) and whether any parts of it can be simplified.\n\n- Providing more qualitative examples, and perhaps a manual evaluation of the generated critiques, would help provide insight into what strategies the critics have learned and whether these critiques are interpretable.", "questions_for_authors": "- How do you think about the CTRL framework being applied to automate the evaluation of tasks that are not as simple as code ?\n\n- Can you describe the sensitivity of your method to the parameters in GRPO? Do you observe any significant performance degradation if these parameters are changed?\n\n- In addition to improving the pass rate, have you performed a qualitative assessment of the quality of the generated comments?\nFor example, how well do these comments agree with human debugging strategies?\n\n- How robust is your approach if execution feedback (i.e., test cases) is noisy or incomplete?\nHave you tested situations where the feedback could be misleading?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DApNUixTrL", "reviewer": "Reviewer_aMwi", "summary": "This paper introduces CTRL (Critic Training via Reinforcement Learning), a framework designed to train critic models for iterative refinement in code generation tasks. The authors propose a two-stage pipeline: supervised fine-tuning (SFT) using execution-guided critique synthesis and reinforcement learning (RL) with Group Relative Policy Optimization (GRPO). CTRL decouples the critic model from the task-performing model, enabling it to provide actionable feedback that improves solution quality without human supervision. Experimental results on multiple programming benchmarks demonstrate significant improvements in pass rates, reduced compounding errors, and generalization capabilities across both base and stronger generator models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback! We appreciate your recognition of our paper’s strengths, especially the `novel combination` of supervised fine-tuning and reinforcement learning for training critics and the `robust experimental design` supporting our claims. Below, we address your specific concerns in detail:\n\n**Scalability to broader tasks beyond code generation**:\n\nWe agree that demonstrating generalization beyond code generation would strengthen our paper. We have conducted **additional experiments** on IFEval [1], a benchmark commonly used for evaluating alignment capabilities [2]. Our results show that, while not explicitly trained on instruction-following tasks, CTRL improves performance on alignment tasks through iterative refinement, demonstrating our approach can generalize to tasks beyond code generation. We will incorporate these findings in our final version to better illustrate CTRL's broader applicability.\n\n[1] Zhou J, Lu T, Mishra S, et al. Instruction-following evaluation for large language models\n\n[2] Yang A, Yang B, Zhang B, et al. Qwen2. 5 technical report\n\n**Sandbox reliance of our method**:\n\nWhile we leverage sandbox execution during training, CTRL can be adapted to tasks without such verification tools through 1) learned reward models for tasks like safety; and 2) reference-based evaluation for tasks with reference responses (e.g., translation) using metrics like ROUGE or BLEU.\n\nIn fact, our framework is agnostic to the specific reward mechanism as long as it can distinguish between successful and unsuccessful revisions.\n\n**Ablation of components (e.g., SFT vs. RL)**:\n\nTables 1 and 2 provide an initial component analysis comparing CTRL-SFT and full CTRL performance. Together with our **additional experimental results** on Pass@5 performance in the [anonymous link](https://anonymous.4open.science/r/icml25-rebuttal/anonymous.pdf), Table 2, we observe that:\n- SFT improves discrimination (F1: 61.19% → 68.55%) and establishes the critique format\n- RL significantly improves single-turn critique-revision (Pass@1: 8.36% → 11.76%) while reducing regression ($\\Delta_\\downarrow$: 3.03% → 0.85%)\n- For multi-sample scenarios, SFT shows larger gains in Pass@5, while RL improves single-sample effectiveness\n\nThis demonstrates that while SFT provides initial improvement in critique format and discrimination, RL training is crucial for generating feedback that leads to successful revisions.\n\n**Essential references not discussed**:\n\nWe thank the reviewer for suggesting highly relevant papers we overlooked. We will include and discuss these works in our final version:\n\n- While CodeDPO improves code generation through DPO with self-generated validation, our work differs by specifically training a critic model to provide human-readable feedback. Besides, CodeDPO relies on larger models for data generation, whereas our approach requires only self-generated critiques for critic training.\n- MultiCritique shares our goal of training critic models but utilizes a multi-agent framework with GPT-4 for meta-critique. In contrast, CTRL achieves strong performance without requiring access to more powerful models during training. Besides, our method enjoys the simplicity of having one critic model instead of four different models.\n- While RLEF uses execution feedback to improve generator models directly, we focus on enhancing the feedback capability of LLMs. In this regard, our approaches are complementary - RLEF trains better generators, while CTRL trains better critics.\n\n**Contributions relative to recent advancements in generative reward models and self-correction**:\n\nWe discuss our position relative to generative reward models in Appendix B, Table 6. Specifically, our approach uniquely **unifies discrimination and refinement** by generating actionable critiques **without direct human supervision**. Our work differs from recent self-correction approaches:\n- SCoRe [3] employs multi-turn online RL to improve self-correction, but focuses on the generator directly correcting its outputs. CTRL decouples critique and generation, allowing specialized training of the critic and demonstrating signs of scalable oversight.\n\n- GLoRe [4] decomposes refinement into when, where, and how to refine. However, they focus on training both global and local refinement models separately, while CTRL trains a specialized critic model that can identify errors and provide actionable feedback to any generator.\n\n[3] Kumar A, Zhuang V, Agarwal R, et al. Training language models to self-correct via reinforcement learning\n\n[4] Havrilla A, Raparthy S, Nalmpantis C, et al. Glore: When, where, and how to improve llm reasoning via global and local refinements\n\n\nWe again appreciate your thoughtful review and will incorporate your suggestions to strengthen the final version of our paper.", "claims_and_evidence": "The claims made in the paper are generally well-supported by experimental results. The authors demonstrate that CTRL-trained critics improve pass rates, reduce error compounding, and generalize across different generator models and benchmarks. However, some claims, such as the scalability of CTRL to broader tasks beyond code generation, are only indirectly supported and lack extensive empirical evidence. Additional validation on more diverse domains could strengthen these claims.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are appropriate for the problem. The use of programming benchmarks like CodeContests, LiveCodeBench, and MBPP+ ensures that the evaluation is rigorous and relevant to the task of code generation. The iterative critique-revision process and the reliance on execution feedback are well-aligned with the problem's requirements. However, the reliance on sandbox execution environments might limit scalability to tasks without such clear evaluation metrics.", "theoretical_claims": "I did not verify all mathematical details rigorously, and some minor steps in the derivations (e.g., bias analysis in Equation 2) could benefit from additional clarification. While the claims are likely correct, their presentation could be more transparent for broader accessibility.", "experimental_designs_or_analyses": "The experimental design is robust, with clear comparisons between CTRL and baseline methods. The use of multiple benchmarks and generator models adds credibility to the results. However, there are some limitations, such as the lack of ablation studies to isolate the contributions of each component (e.g., SFT vs. RL). Additionally, while the authors analyze error compounding and scalability, the experiments are primarily focused on code generation, limiting generalizability to other domains.", "relation_to_broader_scientific_literature": "The paper builds upon prior work in self-improvement of large language models, self-critique, and reinforcement learning for feedback generation. It extends these ideas by introducing a scalable framework that leverages execution feedback and RL. The weak-to-strong generalization phenomenon observed in CTRL aligns with findings in scalable oversight and weak supervision. However, the paper could better contextualize its contributions relative to recent advancements in generative reward models and self-correction in LLMs.", "essential_references_not_discussed": "Some highly related works are not discussed:\n\n1. CodeDPO: Aligning Code Models with Self Generated and Verified Source Code\n2. Training Language Model to Critique with Multi-agent Feedback\n3. RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper presents a novel combination of supervised fine-tuning and reinforcement learning for training critics.\nThe experimental results demonstrate significant improvements over baselines, especially in reducing error compounding and enabling multi-turn critique-revision.\nThe framework's ability to generalize across generator models and tasks is a notable strength.\n\nWeaknesses:\n\nThis paper only focus on the code generation task, leaving alone other tasks especially the open-domain tasks, like summarization, translation, alignment.", "other_comments_or_suggestions": "No", "questions_for_authors": "No questions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes CTRL (critic training via reinforce learning) for code generation, a technique that learns a critique model and performs iterative refinement when paired with a generation model. To train the model, this work proposes two stages, the SFT stage an RL stage with the GRPO algorithm. The reviewers agree that the method is shown to yield solid improvements with comprehensive experiments. There are some concerns around the train/test overlap with the use of the TACO dataset, as well as detailed ablations on the improvements from the SFT and RL stages, which are adequately resolved during the rebuttal phase. However, the main remaining shared concern of this work is the lack of novelty, as methods like iterative refinement, SFT + RL (GRPO) tuning, and learning of critic models are all well-known methods not only for code generation but for many other tasks. Adding more discussions of the missing related works that the reviewers mentioned can also better help position this paper.
Decision: Accept (poster)