Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learning
Authors: Chi Ruan
Dongfu Jiang
Yubo Wang
Wenhu Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Critique Reinforcement Learning
Reinforcement Learning
Critique Fine-Tuning
Large Language Models for Code Generation
Test-Time Scaling
Primary area: reinforcement learning

Abstract: Reinforcement Learning (RL) has emerged as a popular training paradigm, particularly when paired with reasoning models. While effective, it primarily focuses on generating responses and lacks mechanisms to explicitly foster critique or reflection. Several recent studies, like Critique-Fine-Tuning (CFT) and Critique-Guided-Distillation (CGD) have shown the benefits of explicitly teaching LLMs how to critique. Motivated by them, we propose Critique Reinforcement Learning (CRL), where the model is tasked with generating a critique for a given (question, solution) pair. The reward is determined solely by whether the final judgment label $c \in \{\texttt{True}, \texttt{False}\}$ of the generated critique aligns with the ground-truth judgment $c^*$. Building on this point, we introduce Critique-Coder, which is trained on a hybrid of RL and CRL by substituting 20\% of the standard RL data with CRL data. We fine-tune multiple models (Critique-Coder) and evaluate them on different benchmarks to show their advantages over RL-only models.  We show that Critique-Coder consistently outperforms RL-only baselines on all the evaluated benchmarks. Notably, our Critique-Coder-8B can reach over 60\% on LiveCodeBench (v5), outperforming other reasoning models like DeepCoder-14B and GPT-o1. 
Beyond code generation, Critique-Coder also demonstrates enhanced general reasoning abilities, as evidenced by its better performance on logic reasoning tasks from the BBEH dataset. This indicates that the application of CRL on coding datasets enhances general reasoning and critique abilities, which are transferable across a broad range of tasks. Hence, we believe that CRL works as a great complement to standard RL for LLM reasoning.

Published as a conference paper at ICLR 2026
CRITIQUE-CODER: ENHANCING CODER MODELS BY CRITIQUE REINFORCEMENT LEARNING
Chi Ruan"
Dongfu Jiang!.3 Yubo Wang Wenhu Chen 1:2
University Waterloo
Vector Institute
cruan059 Frouan09euottaww.ccm wenhcchenduwattrrloo.coo
attps://tiger-al-lla http://..aaaaaaaa.........ooo.a. wtt......o/crtiiggeeeoooo
ABSTRACT
Reinforcement Learning (RL) has emerged as a popular training paradigm, par- ticularly when paired with reasoning models. While effective, it primarily focuses on generating responses and lacks mechanisms to explicitly foster critique or re- flection. Several recent studies, like Critique-Fing Saeeue--Fie--Tuiiiiii (CFT) and Critique- Guided-Distllation (CGD) have shown the benefits of explicitly teaching LLMs how critique. Motivated by them, propose Critique Reinforcement Learning (CRL), where the model is tasked with generating critique for given (question, solution) pair. The reward is determined solely by whether the final judgment la- bel c E (True, False] of the generated critique aligns with the ground-truth judgment e" Building on this point, we introduce trained on hybrid of RL and CRL by substituting 20% of the standard RL data with CRL data. We fine-tune multiple models (CRITIQUE-CODER) and evalu- ate them on different benchmarks to show their advantages over RL-only mod- els. We show that CRITIQUE- CODER consistently outperforms RL-only baselines all the evaluated benchmarks. Notably, CRITIQUE CONDUUE-CDER-8ER over 60% on LiveCodeBench (v5), outperforming other reasoning models like DeepCoder- 14B and GPT-o1. Beyond code generation, CRITIQUE-CODER also demonstrates enhanced general reasoning abilities, as evidenced by its better per- formance logic reasoning tasks from the BBEH dataset. This indicates that the application of CRL on coding datasets enhances general reasoning and critique abilities, which are transferable across broad range of tasks. Hence, we believe that CRL works as great complement to standard RL for LLM reasoning.
which is
reach
LiveCodeBench (V5)
Aider-Polyglot
EvalPlus
Ozm
145
35.6
93.0
67.5
86.
25
34.8
25
50-
6.
Owend-00
Owen3
Owerd 48
Owen3-80
Qwen3-48
Owen3-UB
Figure 1: 'Te effectiveness of our CRITIQEE-CODER trained with combination CRL and RI. data, compared to baselines and models trained solely RL data, with both training and eval- uation conducted under the think mode setting. EvalPlus denotes the average 4 benchmarks: HumanEval, MBPP, and their corresponding plus version.
INTRODUCTION
Recent breakthroughs in complex reasoning across code generation, mathematical problem-solving, and logical deduction have been driven by large language models such OpenAl's ol-o4 (Jaech et al., 2024a). DeepSeek-RI R1 (Guo et al. 2025). and Kimi-K1.5 (Team et al., (025). key fac- tor behind these advances is the combination of reinforcement learning (RL) with chain-of-thought

---

Published as a conference paper at ICLR 2026
(CoT) (Wei et al., Building on this foundation, research has increasingly focused on scaling reasoning abilities in code generation. For example, AceCoder (Zeng et al. 2025), HardTests (He et al.) 2025), and Kod- Coder (Xu et al. 2025) developed automated large-scale code data generation pipelines and applied RL using reward models and test cases pass rewards, achieving notable performance gains. SWE- RL (Wei et al.] 20255) pioneered the scaling of RL-driven LLM reasoning for real-world software engineering, leveraging an efficient rule-based reward system.
2022 which enables models to iteratively refine intermediate reasoning steps.
Standard reinforcement learning with verifiable reward (RLVR) has shown strong capabilities in improving models' problem-solving abilities. However, this paradigm can hardly elicit the in- ternal critique or reflection behavior on existing solutions. Recently, there has been a line of
work (Wang et af., 2025c; Gou et al., 2024 Xi et al., 2024, Kapusuzogiu et al. 2025 Wang et al. 2025a Tang et al., (2224) aiming to explicitly teach LLMMs to critique to unleash their reasoning ability. Inspired by this, we propose new learning paradigm, Critique Reinforcement Learning (CRL), which incorporates this critique mechanism into RL and explicitly rewards models for ac- curate reflection. CRL not only optimizes problem-solving skills but also explicitly incentivizes its critique abilities via rewards for whether it can correctly judge response's correctness. Spe- cially, the model is prompted with a question-solution pair [q: to predict a binarized judgment label E (True, Fa 1 se), which is compared with the annotated label c" to derive binary (ver- ifiable) reward. This is in contrast to the standard RLVR algorithm, which incentivizes the model to predict correct solution s given query q. as shown in Fiiuure 2 Different from CFT (Wang iet al., (2025c), CRL incentivizes the model based on its self-generated judgment e instead of using teacher-provided critique traces.
Based on the CRL paradigm, we develop CRITIQUE-CODER, a model trained to generate both high- quality coding solutions and critiques on existing solutions. We conducted a series of experiments with the GRPO algorithm (Shao et al. 2024). Specifically, we train Q QWEN3-4B and QWEN3- )025) on the filtered rStar seed dataset (Liu et al. 2025) using a hybrid frame- work that unifies CRI. and standard RLVR. This hybrid approach enables the model to integrate the strengths of both paradigms: CRL helps the model develop critical thinking and reasoning abilitics, while RLVR focuses on enhancing its problem-solving performance. We further adopted an iterative context lengthening approach Luo et al. (2025). where the training context length is extended from 16k to 32k tokens to better leverage the potential of long reasoning chains. By iteratively increasing the context length, the model first develops reasoning skills on shorter contexts, which can then be applied to longer ones.
8B (Yang et al.
BRITIQUR-CODER present consistent improvements across multiple benchmarks, illustrated in Fig- ure From QWEN3-4B, our CRITIQUE- achieves 59.0 accuracy LiveCodeBench (v5) (Jain et al. 2024). yielding +4.8 points over the base model and +2.4 points over the RL-only variant. Remarkably, it even surpasses QWEN3-8B by +1.5 points. On QWEN3-8B, CRITIQUE- CODER reaches 35.6 points on Aider-Polyglot, +7.2 points higher than baseline. It also reaches 60.8 points on LiveCodeBench (v5), which outperforms other reasoning models like DeepCoder- 14B (Luo et al. 2025) and GPT-ol (Jaech et al. 2024b). This showcases the effectiveness of CRL training. Furthermore, results on the logical reasoning benchmark BIG-Bench Extra Hard (Kazemi achieves strong transferable reasoning ability, sur-
et al. 2025) demonstrate that CRITIQUE-CODa passing both baseline and RL-trained models and yielding a +6.1 improvement over the base model. We also find that CRL is more effectively utilized as complement to RL rather than serving as an alternative. This is because CRL training primarily focuses on critiquing question-solution pairs without generating actual solutions. Our ablation study in Table 5) confirms 20% mix ratio best practice.
In summary, we introduce CRL, a novel reinforcement learning (RL) training framework that in- corporates critique learning within the RL paradigm. This novel learning approach enhances the model's critique and reasoning abilities, addressing the lack of critique and reflection incentives typically found in standard RL frameworks. Building on this foundation, we introduce CRITIQUE- CODER, a model combining CRL and RL leverage the strengths of both. CRL fosters criti- cal thinking and reasoning, while RL focuses on optimizing problem-solving Compared to base- line models and those trained exclusively with RL, SEITIQUE-CODER shows superior performance across coding datasets of varying difficulty. Furthermore, the model demonstrates transferable gen- eral reasoning abilities, as evidenced by its strong performance on logic reasoning benchmarks.
2

---

Published as a conference paper at ICLR 2026
2 METHOD
This section introduces the training process of CRITIQUE- CODER, which combines standard rein- forcement learning and critique reinforcement learning for code generation. We first present the problem formulation and optimization method, followed by dataset construction and the overall training procedure.
Standard Reinforeement Learning
Code Comments
Solutiors
Acsuracy
Question
Solutions
Tests
Generate Solutions
Reward
Policy Update
RL+CRL
RL+GRL
Coding LLM Policy
GRPO
dStarCoder Seed
Thinking Length
Code Functions
Reward
Filter Slow Tests
Question
Judggement Solution
Correct Judge?
Solution
Filtered Test Cases
RL RL+CRL
RL AL*CRL
Critique Reinforcement Learning
Comparison between CRL and Standard RL. Standard RL generates solutions based on input questions and evaluates them by executing test cases, while CRL critiques the solution for the paired question and compares the resulting conclusion with the GT determine its correctness. Experiment shows that RL+CRL can improve not only accuracy, but also the code quality.
Figure 2:
2.1 PRELIMINARY
Problem Definition. CRITIQUE-CODDI CODER incorporates two complementary training frameworks for LLMs. The first follows the standard RL setting: given a question q, the policy нe samples n candidate solutions (s:) ) each 8; is evaluated the annotated test cases TT to compute its pass rate, which serves as the reward signal Raçi-
To complement this solution-level feedback, introduce Critique Reinforcement Learning (CRL), which provides binary correction signals on question-solution pairs. Specially, given an annotated dataset D {[gs sk), c ) where each pair ([y: s]) consists of question and solution with an associated binary judgment label c* E 10 the policy me is trained to generate n predictions {<) indicating whether s satisfies the requirement posed by q. The reward Ret, is derived from the comparison between and c*
Finally, two reward signals RA, from RL and Ren, from CRL combined together update policy parameters 9 using GRPO. This unified optimization enables the model to benefit from both critique-guided learning and task-oriented learning, fostering more critical and reflective learning.
Group Relative Policy Optimization (GRPO). We now detail GRPO (Shao et al. 2024). the op- timization algorithm used to update model parameters. In contrast to PPO (Schulman et al. 2017). GRPO enhances performance by leveraging relative performance based updates, which yield more stable and efficient policy refinement. The formal definition GRPO is provided below:
G
lol min (p..A.. 1 cc,1+,),,, (IKL(Tol(TTee)
J(0) E
Σ
G
0:
(1)
To To(0,, 0c,es)
where Pi.
K0wa (0i, Ir, oi,ct)
In the above equation, Ps. denotes the probability ratio of generating output o,, under the new policy me and old policy Tea. A. represents the calculated advantage within each output group, and DxL. regularizes the optimization by measuring the divergence between Ba and the reference policy aa which can prevent the policy from drifting too far away.
3

---

Published as a conference paper at ICLR 2026
Table 1: Dataset statistics of rStar-Coder seed dataset before and after test-case filtering to save the verification time.
Table 2: Dataset difficulty statistics
ing QWEN3-4B (Thinking) with temper- ature=0.6, top-p=0.95, and top-k=20
Metric
Before
After
Metric
Value
Num of Questions
29,365
23,069
Pass @
43.72%
Avg Test Cases
87
24
Pass Pass@4
49.05%
Median Test Cases
48
30
52.98%
Avg Test Case Input Chars
96,208
40
Avg Tokens Solution
13,732
In our training scenario, the policy input be either single question from RL ques- tion-solution pair ([g; s|) from CRL. These two input modalities give rise to distinct reward signals, solution-level rewards Rais and critique-level rewards Rat, Both signals are aggregated in the computation of the advantage Ar,, which makes the GRPO update in our framework fundamentally different from standard RL: the advantage estimation is jointly shaped by task outcomes and critique guidance, allowing the policy align with both execution correctness and reflective judgment.
2.2 DATASET CONSTRUCTION
To evaluate the efficacy CRL, the first step is to build a reliable CRL dataset. The construction process is detailed in the following steps.
construct our CRL dataset from human-seeded RL dataset of rStar- Coder (Liu et al. 2025), which contains a large number of test cases collected from both human- written and verified generated data. To generate these test cases, AStar-Coder employs utility func- tions to generate test case inputs across wide range of scales, reaching up to 10s for challenging cases. As result, many questions in the original dataset include an excessive number of test cases (often exceeding 100 per problem), and some individual cases extremely long (over 10,000 to- kens), Such characteristics substantially increase verification time during RL training. To improve efficiency and consistency, randomly sampling 30 cases for each problem. Tablle I reports dataset statistics before and after filtering, showing a significant reduction both test case length and volume. Specifically, the av- erage input characters decrease from 96,208 to 40, and the average number of cases drops from 87 24. This reduction greatly shortens test case evaluation time during training, resulting in a more efficient learning process. To assess dataset difficulty, we evaluate QWEN3-4B (Yang et al.) 2025) on the filtered dataset, shown Table 2 The model achieves 43.72% at Pass@ and 52.9 at @ass@4, indicating moderate difficulty solvlesolvable in part, yet leaving significant headroom for further progress. This makes the dataset well-suited for RL training under the GRPO algorithm, where advantage is computed within groups.
RL Dataset Selection.
filter the data by discarding test cases longer than 200 tokens and
Question:
Question:
Filter
Q
Inference
You are an experienced Codeforces user
You are experienced
Solution:
Solution:
import bisect def main:
import bisect def main:
Judgment: True
Original Dataset
Test Case Execution
Dataset
Candidate Solution
CRL Data
Figure 3:
Critique data generation. This process involves generating candidate solutions and anno- tating their judgment in the CRL. dataset based on the pass rate over test cases.
Critique Data Generation. Figure 3 illustrates the critique data generation workflow. For each problem, we prompt - ------B----SSTRUUCE Yang al.) 2025) to generate out- puts, from which we extract code blocks as candidate solutions. Despite its larger parameter scale, QWEN3-CODER-30--AA53
lacks explicit reasoning capabilities, and its effec-
4

---

Published as a conference paper at ICLR 2026
tive problem-solving performance is not stronger than that of smaller reasoning-enabled models such as QWEN3-4B (Thinking). Empirically, QWEN3-CODER 30--A33--------- -3B-INSTRUCT achieves a pass @ 1 of 32.20% our dataset, compared to 43.72% for QWEN3-4B (Thinking). Owing this relatively weak performance, the critique data generation process does not constitute a distillation setup, as the data are not produced by stronger teacher model. Empty code blocks are discarded to ensure that only valid programs are retained for evaluation. Each candidate solution is then executed on the test cases from the filtered dataset, and its pass rate computed determine its judgment. A practical challenge is that certain test cases exhibit excessively long execution times, which may cause timeouts and lead the erroneous classification of correct solutions failures. To relax this constraint, we adopt pass rate threshold of 80%: candidate solutions are labeled as True if their test pass rate exceeds this percentage, and as Fa se otherwise.
Hybrid Data Integration. In our case, training exclusively on critique-oriented data biases the model toward evaluative behaviors, encouraging lutions rather than directly producing task-oriented outputs, which in turn degrades its ability to generate complete answers in evaluation tasks. Standard RL data, by contrast, explicitly reinforces end-to-end solution generation through task-level rewards but does not expose the model to reflec- tive or critique-base based reasoning patterns. To mitigate this imbalance, we construct a hybrid dataset that combines both CRL and standard RL data, allowing the model to jointly learn evaluative judg- ment and direct solution synthesis within a unified training process. Concretely, we randomly assign 20% of the data from the dataset to be CRL data, with the remaining 80% consisting of standard RL data. Such a configuration not only mitigates the risk of format shift caused by overexposure to critique-style supervision but also improves the robustness of the learning process by exploiting the complementary strengths of CRL and RL training paradigms.
focus on judging or analyzing candidate so-
2.3 TRAINING
Training procedure. Algorithm [ presents the training procedure of CRITIQUE-CODER, which integrates CRL and standard RL within unified framework. The policy model is initialized from pretrained checkpoint Th... and trained on hybrid dataset containing both CRL samples with judgment labels and RI samples with associated test cases. At each training step, the policy sam- ples G candidate outputs for each instance under either the CRL or RL. setting. For CRL data, the model produces multiple critique predictions, parses the judgment from the Aconclus field, and computes rewards by comparing the predicted judgments with the ground-truth labels. For RL data, the model generates multiple solution candidates, extracts the code blocks enclosed by [code]
}
and evaluates them against the provided test cases to obtain inccutiin-basse rewards. These rewards are then transformed into advantage estimates and jointly used to update the policy parameters through GRPO.
Algorithm Training procedure of CRITIQUE- CODER
Input dataset D {[g; s;), + U Ui(g, ((gg,tt))
1 N3 policy To
1: Initialize policy model
mea..
2: for each step do
3: Sample a batch B C
4: for each data instance d E B do
5: if d ((/; s),e*) c))c*)
D CRL data with judgment then
6:
Sample G outputs {o Te((;;)) s))
7:
Parse each o, extract judgment e inside \conclusion
8:
Compute reward Ra,,i e²) for each ç;
9:
else d (q.t)
b RL data with test cases then
10:
Sample G outputs {o:)_] ~ TT(2)
11:
Parse each o, to extract solution 8; enclosed by '. [code] .. Evaluate on test cases i, obtain reward (s,,
12:
13:
end
14:
end for
15:
Compute A,, from reward R3, where R, Res. (G; c*) for CRL or Ra, t) for RL.
16:
Update the policy model my with GRPO Equation
17:
end for
5

---

Published as a conference paper at ICLR 2026
Reward function. As specified in Algorithm rewards computed from two data sources: CRL and RL. For CRL samples, the model is prompted to store the final judgment in /conclus ion () from which the predicted label css extracted. A reward of is assigned if e matches the ground truth c"; otherwise, including the case where the prediction is missing, the reward is O. For RI. samples, the reward is defined the pass rate across test cases. Formally,
Ronle, c²) =
if cf 20,
[1,
Rn(s.T) K N
(0, otherwise,"
where T is the set of N test cases and K denotes the number of cases successfully solved by the model's output s. Thus Rς E [0, Il, with larger values indicating more reliable solutions. At the batch level, each instance receives reward according its data type:
(e)( if E BERL:
R²
(3)
Ra(s; Ti), if BRL.
Here BERL and BRL denote the CRL and RL subsets within the batch, respectively.
EXPERIMENT
This section presents comprehensive empirical evaluation demonstrate the effectiveness of CRITIQUE- CODER.. We report training and evaluation settings, compare against baseline and RL- trained models on multiple coding benchmarks, and further analyze transferability, test-time scaling, and ablation results.
3.1 TRAINING SETUP
We conducted experiments on two models, QWEN3-4B and QWEN3-8B (Yang et al. 2025). in thinking mode. Following two-phase training strategy similar to DeepCoder (Luo et al. 2025), set the maximum response length to 16k in the first phase and increase it to 32k once the rewards have stabilized. During training, two rule-based rewards employed, each tailored to different data type. For CRL data, the reward 1.0 the prediction matches the ground truth (GT) and 0.0 otherwise; additionally, during the 16k phase, this reward scaled by factor of 0.8 reduce its dominance relative to RL signals. For RL data, the reward corresponds to the pass rate over test cases, ranging from 0.0 to 1.0. For standard RL training, we adopt the thinking prompt used in the Qwen3 paper on LiveCodeBench, while the CRL training prompt is provided in Appendix [A] Throughout training, we apply the GRPO algorithm (Shao et al. 2024), which provides improved stability and efficiency compared to PPO (Schulman et al.) (017) The hyperparameters are set as follows: a batch size of 128, learning rate of 1c-6, and sampled outputs per prompt. To encourage exploration while stabilizing entropy, the clipping ratio is asymmetric, with an upper bound of 0.3 and lower bound of 0.2. We trained the model on the entire dataset for one epoch, and selected the best-performing checkpoint using the LiveCodeBench (y5) the validation set.
3.2
EVALUATION SETUP
To evaluate and compare our training results, we utilized four different benchmarks: EvalPlus (Liu al. 2023) (an average of HumanEval (Chen al.] 202T), HumanEEvl+, MBPP (Austin et al. 2021 and MBPP+), BigCodeBench- Bench-nennn- (Zhuo et al. 2024), Aider-Polyglot (Aider (2024), and LiveCodeBench (v5, 2024.10 1ee.1102222..02 (Jain al.) (024). These benchmarks cover a diverse range
of coding tasks, enabling a comprehensive assessment of the model's code generation ability.
For the sampling configuration, we follow the thinking mode settings reported in the original Qwen3 paper (Yang et al.] (025), with temperature of 0.6, a top-p of 0.95, a top-k of 20, and maximum output length of 32,768 tokens. The same configuration applied consistently across all evalua- tion tasks. For LiveCodeBench, mode to ensure consistency. We further compare our model with several strong coding baselines, including DeepSeek-RR -1-tt-Rii-----11- (Guo al.) 2025), DeepCoder (Luo al. 2025), DeepSeek- V2.5 (DeepSeek-Al) 2024), and GPT-ol (Jaech et al.] (024b) with high reasoning eftort, and report results from their official publications recommended evaluation settings.
adopt the official evaluation prompt used in Qwen3 thinking

---

Published as a conference paper at ICLR 2026
Table 3: CRITIQUE-CODER performance compared with baseline and models trained with standard RL. The RL training was conducted on the filtered rStar-Coder seed dataset, and the CRL training was carried out by converting 20% of the data into CRL for fair comparison.
TigCodeBenche
LiveCodeBench v5
Model
EvalPlus
Aider-Polyglot
AVG
Full
Hard
AceCoder-7B
82.7
43.3
19.6
DeepSeck-R SeopScck-11-i R1-Distill-144
82.4
38.1
20.9
18.6
53.0
42.6
DeepCoder- 14B
85.3
38.2
18.2
18.4
60.6
44.1
DeepSeek- DeepSeck-V2.522538
83.8
48.9
27.0
17.8
42.6
44.0
GPT-ol
50.4
28.4
61.7
59.5
57.7
Baseline Qwen3-4B (Thinking)
Baseline
85.2
42.0
20.9
21.8
54.2
44.8
Qwen3-4B-RL
84.9
40.6
23.0
23.6
56.6
45.7
CRITIQUE-COMe CODDER
86.5
43.1
23.0
24.4
59.0
47.2
A curs-Baselinn)
+1.3
+1.1
+2.1
+2.6
+4.8
+2.4
Baseline Qwen3-8B (Thinking)
Baseline
85.8
44.6
23.6
28.4
57.5
48.0
Qwen3-8B-RR RL
86.2
44.5
24.3
34.5
59.6
49.8
CRITIQUE- CRITIQUE-CODER
87.7
46.6
27.0
35.6
60.8
51.5
À (Ours-Baseline)
+1.9
+2.0
+3.4
+7.2
+3.3
+3.5
3.3
MAIN RESULTS
Gains Base Models. Compared with the base models, CRITIQUE-CODER leads consistent and notable improvements across benchmarks of varying difficulty levels and evaluation protocols. On QWEN3-4B, for example, the LiveCodeBench score of BRITIQUE-CODER increases from 54.2 to 59.0, yielding substantial gain of +4.8 and even surpassing the larger QWEN3-88 -B baseline by +1.5 points. On the Aider-Polyglot benchmark, which evaluates multi-language programming ability. CRITIQUE-CODER achieves +7.2 improvement over the QWEN3-8B baseline, despite being trained solely on Python data using CRL. Importantly, these gains are consistently reflected across all evaluated benchmarks and model scales, and also translate into higher overall average scores (+2.4 for 4B and +3.5 for 8B), indicating that the improvements are broad-spectrum rather than limited to specific tasks datasets.
Advantages over RL-trained Models. Under identical datasets and training configurations, re- placing part of the RL data with CRL consistently yields superior results across all benchmarks. On QWEN3-4B, CRITIQUE- CODER exceeds the Qwen3-4B-Ri by +2.4 points on LiveCodeBench and improves the overall benchmark average by +1.5 points. On QWEN3-8B, outperforms the Qwen3- 8B-RL counterpart by +2.7 points BigCodeBench--arc contributing to average gain of +1.7 points across all benchmarks. These findings highlight that CRL brings complementary benefits RL, enabling CRITIQUE-CODER ther analyze the test outputs of CRL and standard RL on LiveCodeBench, as illustrated in [iiiuree 4 The results show that CRL generates longer reasoning traces the think blocks. indicating more extensive deliberation and reflection, and confirming that CRL indeed enhances the model's 'ss reason- ing and critique capabilities. also incorporates markedly more explanatory comments within the generated code, indicating stronger tendencies toward self-explanation.
achieve more robust and consistent improvements. We fur-
Comparison with Frontier Models. Despite their relatively small parameter sizes, our mod- els demonstrate strong absolute performance when compared with frontier systems. CRITIQUE- CODER-4B significantly outperforms DeepCoder- 14B (Luo et al.) (025) across all evaluated bench- marks while using only 28% of its parameters, highlighting the parameter efficiency of our approach. CRITIQUE- MATQQUE-COEER CODER- 8B remains competitive with other strong models and only lags behind GPT- ol (Jacch et al. 2024b) on Aider-Polyglot, which can be largely attributed to its lack of explicit optimization for non Python languages. On EvalPlus, sive score of 86.5, closely trailing GPT-o1, while CRITIQUE CODER-S 8B further improves this result to 87.7, demonstrating that our method scales effectively with model size.
MATTQUEECCDDER
achieves an impres-

---

Published as a conference paper at ICLR 2026
Thinking Black Usage
Code Qualiity Comparison
LiveCodeBench (v5)
118814
2.30
250
20000
Artoag Thzaking Longt
220
3.50
2000
150
1400
Teat Time Scaling Bleations (1-6)
2000
06-
Starting Point
06
CiL
3395
Code Qaalls MOUK
Tokens per Question
Figure 4: Analysis of the generations on the Live- CodeBench (v5) problems. Results show that CRL can elicit better reasoning behavior and coding quality.
Figure 5: Test-time scaling perfor-
mance of CRITIQUE-CODER-DD LiveCodeBench (v5)
Table 4: Performance comparison on four BIG-Bench Extra Hard (BBEH) logic reasoning subtasks
Model
Time Arithmetic
Disambiguation(A
Zebra Puzzles
BoardgameQA
AVG
Baseline Qwen3 (Thinking)
Baseline
40.5
43.3
36.5
66.5
46.7
Qwen3-4B-RI.
45.0
43.3
40.5
66.5
48.8
CRITIQUE-COU CODER
48.5
47.5
44.0
71.0
52.8
A (Ours-Baseline)
+8.0
+4.2
+7.5
+4.5
+6.1
3.4 TRANSFERABLE REASONING ABILITY
To examine whether the critique and reasoning abilities learned by CRITIQUE- CODER extend be- yond coding tasks, we evaluate the model on BIG-Bench Extra Hard (BBEH) logic reasoning benchmark (Kazemi et al. 2025). As shown in Table 4 BRITIQUE-CODER consistently outper- forms both baseline QWEN3-4B and its RL-trained variant across all four reasoning subtasks, achieving average improvement of +6.1 points over the base model. The gains especially pronounced on more structured and constraint-intensin tasks such as Time Arithmetic (+8.0) and Zebra Puzzles (+7.5), which require multi-step logical consistency rather than surface-level pattern matching. Moreover, despite being optimized with reinforcement learning, Qwen3-4B-B shows only limited improvements, whereas BRITIQUE-CODER achieves additional +4.0 average gain and outperforms it by +4.5 points on BoardgameQA. Notably, we use the same inference configu- ration for all models, suggesting that the gains are driven by critique-enhane evaluation- time settings. Overall, these results indicate that critique-enhann enauneeeeenneedd training provides ad- ditional benefits over standard RL optimization and effectively improves reasoning abilities that transfer beyond coding tasks.
training rather than
3.5 TEST TIME SCALING
CODEREE--CCDDD with sequence test-time scaling (budget forcing) (Muen- nighoff et al, 2025). which relaxes the constraint on reasoning tokens at inference to enable longer reasoning trajectories. As shown Figure 5] increasing the test-time computation budget leads to consistent and monotonic improvement on LiveCodeBench (y5): starting from the default setting without test-time scaling (approximately 13k tokens per question and 59.0 accuracy), the model steadily improves the number of scaling iterations increases, reaching an accuracy of 62.0 after four iterations and achieving a +3.0 absolute gain over the no-scaling baseline. Further increases in the reasoning budget yield marginal but stable improvements, indicating diminishing returns. Overall, these results demonstrate that test-time scaling is an effective mechanism for enhancing the performance of compact 4B code model without modifying its parameters. We note that this improvement specific to sequence- based test-time scaling: in contrast, parallel test-time scaling does yield gains, which analyze in detail in subsection 3.7
We further evaluate
8

---

Published as a conference paper at ICLR 2026
Table 5: Impact of different CRI. data proportion. All datasets derived from the filtered rStar- Coder seed dataset, with varying proportions of RL data converted into CRL data.
BigCodeBench-I
LiveCodeBench v5
Model
EvalPlus
Aider-Polyglot
AVG
Full
Hard
Baseline Qwen3-4B (Thinking)
0% of CRL Data
84.9
40.6
23.0
23.6
56.6
45.7
50% of CRL Data
86.5
42.4
22.3
24.0
56.0
46.2
100% of CRL Data
85.2
41.6
17.6
21.3
56.6
44.5
20% of CRL Data (Ours)
86.5
43.1
23.0
24.4
59.0
47.2
3.6 ABLATION STUDY
To study the impact of the CRL data ratio, we conduct an ablation study by replacing different proportions of RL data with CRL data while keeping the total amount of training data and opti- mization settings fixed. All training samples are derived from the same filtered rStar-Coder seed dataset, and CRL data are obtained by converting subset of the original RI. samples following the same procedure. As shown in Table 5] introducing a moderate amount of CRL data consistently improves performance over the pure RL baseline. Among all configurations, the model trained with 20% CRL data achieves the best overall results, increasing the average score from 45.7 to 47.2 (+1.5). This setting yields the strongest tied-best performance most benchmarks, including EvalPlus, BigCodeBench-I (Full), Aider Poler-ooyyglot, and LiveCodeBench v5, demonstrating that CRL effectively complement RL training.
As the CRL ratio increases further, the performance gains gradually diminish. Although the 50% CRL setting remains competitive on average, it shows degradation more challenging subsets such as BigCodeBench- (Hard). When RL data are completely replaced by CRL data, performance drops significantly across multiple benchmarks, resulting baseline. These results suggest that CRL should be used as complementary signal rather than replacement for RL data. We hypothesize that excessive reliance on CRL may bias the model ward judgment-orientee behaviors, introducing a mismatch with inference-time requirements that emphasize long-horizon solution generation, particularly on harder benchmarks. Overall, maintain- ing an appropriate balance between RI and CRL data is crucial, with 20% CRL providing the most favorable trade-off our experiments.
average score (44.5) lower than the
3.7 LIMITATIONS IN SELF-CRITIQUE
Although incorporating CRL substantially improves the model's reasoning and critique-related pabilities during generation, does not enable reliable post-hoe self-critique when only the final answer is available. To examine this limitation, implemented baase parallel test-time scaling on CRITIQUR-CODER prompted to generate 10 candidate solutions per problem. Each solution, together with the original question, was then fed back into the model for critique, with 64 critique samples generated per so- lution. Candidate solutions were scored by the number of True critiques they received, with ties broken by selecting the solution with the shortest critique thinking token length. This procedure did lead to performance improvements.
Specifically, on LiveCodeBench v5, CRITIQUE-CODER
was
This outcome is not specific to our method and is consistent with prior findings (Huang et al. 2023 Valmeekam et al. 2023) showing that large language models cannot reliably evaluate or refine their own final outputs without access to intermediate reasoning trajectories or additional external signals. Importantly, this result does not contradict the effectiveness of CRL. The performance gains from CRL do not arise from post-hoc self-critique on completed answers; instead, CRL improves perfor- mance by internalizing critique-base based judgment during training, enabling more thorough reasoning and refinement before the final answer is produced. Consequently, once the reasoning process is moved and only the final output is provided, the model naturally fails to further critique or improve it, in line with established limitations in the literature.

---

Published as a conference paper at ICLR 2026
4 RELATED WORKS
4.1 CRITIQUE LEARNING
The idea of leveraging critiques for improving model reasoning has been explored in several lines of research. One direction self-correction (Bai et al. 2022 Gou et al.) 2023; Madaan et al.) 2023), where models iteratively evaluate and revise their own outputs. Although such methods are promising, subsequent studies have questioned their robustness and consistency (Huang et al. 22023 Valmeekam et al. (2023). Another line involves reward models (Uesato et al.) 2022 Wang et al. 2023 Lightman et al. 2023), which act as learned evaluators that assign quality scores to either final outputs or intermediate reasoning steps, thereby guiding reinforcement learning to enhance reasoning capabilities. More recently, Critique Fine-Tunine Tuning (CFT) (Wang al. 2025c] b) explicitly trains models to critique candidate solutions, demonstrating improved reasoning ability.
2023 Shinn et al.
Our approach is most related to CFT. Unlike CFT, which directly optimizes the model to imitate critique reasoning process, CRL instead encourages the model to actively explore and learn from the correctness of its final judgments, thereby combining the benefits critique reasoning with reinforcement feedback.
4.2
REINFORCEMENT LEARNING FOR CODE GENERATION
Code generation, a core capability of LLMs, has received considerable attention. CodeRL (Le al., 2022) introduces a pioneering RL framework for code generation, employing an actor-critic archi- tecture to encourage functionally correct outputs. Building on this foundation, PPOCoder (Shojaee et al. 2023) incorporates the PPO algorithm to further stabilize and improve training. Moreover, RLEF (Gehring et al. 2024) advances the paradigm by explicitly leveraging execution feedback during synthesis. More recently, AceCoder (Zeng et al. (2025) proposes scalable pipeline that automatically constructs question-ted testio case pairs from code data to facilitate RL training.
4.3 CHAIN-O--THOUGH REASONING.
Recent advances in reasoning language models (RLLMs) show that extended chain-of-thought (CoT) reasoning substantially improves performance on tasks like coding and mathematics. OpenAI ol (Jaech et al. (2024a) and DeepSeek RI (Guo et all 2025) exemplify this trend by using inference- time scaling, where models iteratively explore and reflect before converging on solution. Building on this, KIMI K1.5 (Team et al.] 2025) simplifies the reinforcement learning framework while in- corporating long-context scaling and enhanced policy optimization, further advancing reasoning efficiency. More recently, Qwen3 (Yang et al. 2025) combines thinking mode for reasoning with non-thinking mode for fast responses, switching between them to balance latency and performance.
5 CONCLUSION
In this paper, introduce Critique Reinforcement Learning (CRL), novel reinforcement learning paradigm that integrates critique learning into the RL. by incorporating feedback on the correctness of critiques predicted by the model. Building this foundation, developed CRITIQUE-CODER, trained on a mix of RL and CRL data. On LiveCodeBench v5, CRITIQUE-CODER- score of 59.0, outperforming the baseline by +4.8 points and the RL-only model by +2.4 points. In addition to coding tasks, CRL also enhances general reasoning ability. On the BBEH logical reasoning benchmark, CRITIQUE- CODER shows substantial improvements, surpassing the baseline and RL-trained models by +6.1 and +4.0 points on average across four subtasks. These results demonstrate that CRL not only boosts critique and reflection abilities over standard RL but also enables these capabilities to extend beyond coding domains. However, ablation studies reveal that training exclusively on CRL data yields poorer performance than RL alone, since CRL focuses on generating critiques rather than task-oriented solutions, leading to a mismatch with evaluation requirements. Therefore, rather than substituting RL, CRL serves as a powerful complement to it. Taken together, our findings demonstrate that CRL enhances standard RL by endowing models with stronger critique and reasoning abilities-capabllitee that manifest not only in coding tasks but also transfer effectively to broader reasoning domains.
achieves a
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Aider. Aider-polyglot benchmark. benchheak https://aider.cm /Ttp:s//aaaee..caa///222///////0 .htm
1 1#the-pol #tthe--polyylloc--becchaarrrrr 2024.
Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoc Le, al. Program synthesis with large language models. arXiv preprint arXiv:2108.0775 2021.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
Mark Chen, Jerry Tworek, Heewoo Jun Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
DeepSeek-AL. Deepseek-v2: A strong, economical, and efficient mixture-of-expprts language
model, 2024.
Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Quentin Carbonneaux, Taco Cohen, and Gabriel Synnaeve. Rlef: Grounding code Ilms in exccution feedback with reinforcement learning. arXiv preprint arXiv:2410.0200 2024.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang. Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint MxXiv:2305.11388 2023.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Nan Duan, Weizhu Chen, et al. Critic: Large language models can self-correct with tooe-interaetive critiquing. The Twelfth International Conference on Learning Representations, 2024.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims 2025.
via reinforcement learning. arXiv preprint arXiv:2501.
Zhongmou He, Yee Man Choi, Kexun Zhang, Jiabao Ji, Junting Zhou, Dejia Xu, Ivan Bercovich, Aidan Zhang, and Lei Li. Hardtests: Synthesizing high-quality test cases for Ilm coding. arXiv preprint arXiv:2505. 24099, 2025.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798, 2023.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint 4rXiv:2412.167222 2024a.
OpenAl: Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed Ei-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum. Angela Jiang, Ashvin Nair. Barret Zoph. Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Eliz- abeth Prochl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter. Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace
11

---

Published as a conference paper at ICLR 2026
Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman, Haiming Bao, Hao Sheng, Hart An- drin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman. Hyung Won Chung, lan Kivlichan, Ian O' Connelll, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, llya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang. Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang, Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kon- draciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu. Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe, Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang. Mingxuan Wang, Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowdhury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Reimar Leike, Renny Hwang, Rhythm Garg. Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sand- hini Agarwal, Santiago Hernandez, Sasha Baker, Scott McKinney, Scottie Yan. Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay, Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang, Taylor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peter- son, Tyna Eloundou, Valerie Qi, Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong. Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojeiech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang. Zheng Shao, Zhuohan Li, and et al. Openai ol system card. Technical Report arXiv:4412 OpenAl, San Francisco, CA, December 2024b. URL [cs.AI].
Míyara,
112222166660 arXiv:2412.16720
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07 079774 2024.
Berkcan Kapusuzoglu, Supriyo Chakraborty, Chia-Hsuan Lee, and Sambit Sahu. Critique- guided distillation: Improving supervised fine-tuning via better distillation. arXiv preprint 1521: 2025.
Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, San- ket Vaibhav Mehta, Lalit K Jain, Virginia Aglictti, Disha Jindal, Peter Chen, et al. Big-bench extra hard. arXiv preprint arXiv:2502. 19187, 2025.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 14:2114--13228 2022.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chat- gpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558 21178221522211 2023.
Yifei Liu, Li Lyna Zhang, Yi Zhu, Bingcheng Dong, Xudong Zhou, Ning Shang, Fan Yang, and Mao Yang. rstar-coder: Scaling competitive code reasoning with a large-scale verified dataset. arXiv preprint arXiv:2505.2222 2025.
Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica.
12

---

Published as a conference paper at ICLR 2026
A fully spenc 14b coder at level. http://wwwwwo..ttpp///e/pttaaaaa out.uuae...clllcccooeerrooroaaaa +=tea3a-ccciuseee488803333333333
Deepcoder:
2025. Notion Blog.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri
Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534 46594, 2023.
Niklas Muennighoff, Zitong Yang. Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, arXiv preprint arXiv:2501. 19393, 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1oXiv:/707.00 17077707776777 2017.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634 865226666,,6- 2023.
Parshin Shojace, Aneesh Jain, Sindhu Tipimeni, and Chandan K Reddy. Exccution-based code generation using deep reinforcement learning. arXiv preprint arXiv:2301.13816 2023.
Zhengyang Tang, Ziniu Li, Zhenyang Xiao, Tian Ding, Ruoyu Sun, Benyou Wang, Dayiheng Liu, Fei Huang, Tianyu Liu, Bowen Yu, al. Self-evolving critique abilities in large language models. In Second Conference on Language Modeling, 2024.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with Ilms. arXiv preprini arXiv:2501. 12599, 2025.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint wxxv:2211.11425.. 2022.
Karthik Valmeckam, Matthew Marquez, and Subbarao Kambhampati. Can large language models really improve by self-critiquing their own plans? arXiv preprint arXiv:2310.08118, 2023.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. sath-shepherd: Verify and reinforce Ilms step-by-step without human annotations. arXiv preprint ar arXiv:2312.00080 2023.
Yubo Wang, Ping Nie, Kai Zou, Lijun Wu, and Wenhu Chen. Unleashing the reasoning potential of pre-trained I1ms by critique fine-tuning on one problem. Proceedings of EMNLP, 2025a.
Yubo Wang. Ping Nie, Kai Zou, Lijun Wu, and Wenhu Chen. Unleashing the reasoning potential of pre-trained Ilms by critique fine-tuning one problem. arXiv preprint arXiv:2506.0322 033295 2025b.
Yubo Wang, Xiang Yue, and Wenhu Chen. Critique fine-tuning: Learning to critique is more effec- tive than learning to imitate. arXiv preprint arXiv:2501.17703, 2025c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing 1lm reasoning via rein- forcement learning on open software evolution. arXiv preprint arXiv:2502. 20..22222.4444 18449, 2025.
Zhiheng Xi, Dingwen Yang, Jixuan Huang, Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shihan Do, Wenyu Zhan, et al. Enhancing I1m reasoning via critique models with test-time and training-time supervision. arXiv preprint 1xXi:2411.,16599 2024.
13

---

Published as a conference paper at ICLR 2026
Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding. arXiv preprint arX1v:2503.02951, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.00388 2025.
Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. Acecoder: Acing coder rl via automated test-case synthesis. arXiv preprint arXiv:2502.01718, 2025.
Terry Yue Zhuo Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Bench- marking code generation with diverse function calls and complex instructions. arXiv preprint 15877, 2024.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A CRL TRAINING PROMPT
We provide the prompt template used for constructing CRL training data.
Table 6: Prompt for constructing CRL training data
You will be given question (problem specification) and submitted solution. Your task is to determine whether the solution is correct and fully satisfies the specification.
Question: {question}
Solution: {solution)
Conclude with (conclusion(T) for correct, |conclusion (F} for wrong.
15


Reviews:
[{"review_id": "NWKfMBfhxx", "reviewer": "Reviewer_jfJs", "summary": "This paper proposes training code generation models on a mixture of two tasks: standard code generation and critique evaluation. The critique task involves judging whether a given solution correctly solves a problem, receiving binary rewards based on judgment accuracy. The authors train Critique-Coder using GRPO on 80% code generation data and 20% critique data, fine-tuning Qwen3-4B and Qwen3-8B models. Results show improvements across coding benchmarks, with Critique-Coder-8B achieving 60.8% on LiveCodeBench (v5). The approach also improves performance on BBEH logical reasoning tasks, suggesting the critique task develops transferable reasoning skills beyond coding.", "questions": "- Why generate critique data from a different 30B model instead of using the training model's own solutions? Did you consider on-the-fly critique generation during training?\n- Have you investigated whether the test-time critique failure is specifically due to the distribution mismatch between training critiques (from 30B model) and test-time critiques (of own solutions)?\n- Can you provide ablations isolating the source of improvements: is it multi-task learning, distillation from the larger model, or genuine critique capability?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Comprehensive evaluation across diverse benchmarks (LiveCodeBench, EvalPlus, Aider-Polyglot, BigCodeBench) demonstrates consistent improvements over single-task training baselines.\n- Evidence of transfer to general reasoning (BBEH benchmark) suggests the critique task develops broadly applicable skills rather than narrow task-specific capabilities.\n- Ablation study provides practical guidance, showing 20% critique data mixing is optimal. Too much critique data (50-100%) degrades performance due to train-test mismatch.", "weaknesses": "- Critique training data is generated using a different model (Qwen3-Coder-30B-A3B-Instruct) rather than the model's own solutions. This creates a fundamental distribution mismatch, the model learns to critique a 30B model's solution distribution, not its own, which likely explains the complete failure of test-time self-critique in Section 3.7.\n- The paper is missing the obvious experiment which would be on-the-fly critique generation where models critique their own on-policy solutions during training. The paper currently takes a shortcut using pre-generated data, that too from a different model. \n- Confusing terminology—\"CRL\" and \"RL\" both refer to GRPO training on different tasks (critique vs. generation). This is fundamentally a  multi-task learning setup rather than a novel training algorithm.\n- Due to lack of proper ablations, it's hard to discern the reasons for the performance boost. The observed improvements could stem from multiple confounds: simple multi-task learning effects, knowledge distillation from the 30B model, or just increased training compute. The paper doesn't isolate the actual source of gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9JdPUmvVgg", "reviewer": "Reviewer_badP", "summary": "This paper presents Critique Reinforcement Learning (CRL), a training paradigm designed to teach large language models to critique solutions rather than merely generate them. Given a (question, solution) pair, the model produces a critique containing a final binary judgment  c∈{True,False}, which is rewarded based on whether it matches the ground-truth correctness label . The authors further propose CRITIQUE-CODER, trained with 80% standard RL data and 20% CRL data. Empirical evaluations on EvalPlus, BigCodeBench, LiveCodeBench v5, and BBEH benchmarks show that CRITIQUE-CODER consistently outperforms RL-only baselines, including DeepCoder-14B and GPT-o1, especially in code reasoning tasks.\nWhile the paper is well-executed and clearly written, its claim of being “verifier-free” and “self-improving” is only partially justified, since CRL fundamentally depends on verifiable signals derived from executable test cases and strong model–generated solutions.", "questions": "1. How would CRL perform if the candidate solutions were generated by a weaker or equal-capacity model?\n\n2. Have you attempted CRL in non-code domains lacking executable verification?\n\n3. Could you provide training curves or reward variance to assess stability under binary rewards?\n\n4. What differences exist between your approach and CFT/CGD when controlling for compute and data volume?\n\n5. Why do self-critique variants fail—does this relate to policy–critic decoupling or distributional mismatch?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Addresses a timely question: how to endow models with critique and reflection abilities.\n\n2. Clear algorithmic presentation (Algorithm 1) and reproducible training setup.\n\n3. Comprehensive experiments across multiple code benchmarks.\n\n4. Demonstrates that mixing 20% CRL data improves robustness and mitigates policy collapse.\n\n5. Writing quality, figures, and clarity of exposition are strong.", "weaknesses": "1. Dependence on verifiable signals and strong model–generated solutions.\nThe ground-truth labels \nare obtained through executable test cases with a pass-rate threshold (80%), and the candidate solutions are generated by a stronger model (Qwen3-30B). Although no external verifier module is used, this still constitutes a verifiable supervision pipeline, limiting applicability to domains where executable feedback is available.\n\n2. Limited differentiation from prior critique-based methods.\nCRL is conceptually close to CFT and CGD, differing mainly in the reinforcement reward design. The absence of direct experimental comparisons leaves its unique contribution ambiguous.\n\n3. Sparse reward and lack of training dynamics analysis.\nThe reward is binary (0/1), yet the paper provides no discussion of variance, convergence stability, or reward sensitivity to the pass-rate threshold.\n\n4. “General reasoning” claims rely on narrow evidence.\nImprovements on the BBEH dataset are encouraging but cover only one reasoning benchmark. There is no evidence of gains in non-verifiable or natural-language reasoning domains.\n\n5. Absence of self-critique capability (acknowledged by authors).\nSection 3.7 explicitly reports that enabling the model to critique its own outputs does not improve performance, suggesting the learned skill is evaluative (“other-critique”) rather than reflective (“self-critique”).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DIBt6zCs0K", "reviewer": "Reviewer_Yx6b", "summary": "This paper introduced critique reinforcement learning (CRL). It is a training paradigm designed to improve models' reasoning and reflective abilities for code generation tasks. It  explicitly trains model to  critique and judge the correctness of generated solutions. \nSo in addition to traditional outcome-based signal, they introduced a new signal regarding the correctness on question-solution paris. The experiments on several code generation tasks show improvements after leveraging critique reinforcment learning.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduced a new perspective which is to leverage critique signals to improvement the performance during reinforcement learning.\n2. Tis work uses test case execution results to automatically generate judgement labels, enabling scalable data creation without human annotation.", "weaknesses": "1. The improvement of 8B model is limited. For example, on LiveCodeBench, it improves 1.2% and on Aider-Polyglot, it improves 1.1%. Setting aside noise and variance, I observe that as the model size increases, the performance gain diminishes. This raises concerns about whether the proposed method can effectively scale up.\n\n2. For CRL process, where are the groundtruth judgement label c* from? From Figure 3 and page 4, it looks to me these judgement labels are from the text case execution and its pass rate. However, during normal RL training, we  have already had such signals from pass rate of each rollout solution. The proposed method just make it happen earlier and separately. I wonder how much gains could obtain from this process and what's the real source of improvement.\n\n3. In the current implementation, how do you aggregate two rewards: reward from RL and reward from CRL? Since multi-objective RL is not very robust, Is there any instability or robustness issue with different aggregation methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PzeqVmXEvN", "reviewer": "Reviewer_qbh7", "summary": "This paper proposes integrating *critique data* into reinforcement learning from verifiable rewards (RLVR) for large language models (LLMs). The critique rewards are binary (1–0), indicating whether the model’s judgment is correct relative to a reference solution. The study focuses on coding tasks, where solutions can be automatically verified using unit tests. Experimental results demonstrate that incorporating critique data leads to significant performance improvements on coding benchmarks such as **LiveCodeBench**.", "questions": "The authors note that\n\n> Although incorporating CRL enhances the model’s reasoning and critique abilities, it does not exhibit self-critique capability.\n\nThey further present an experiment showing that having the model critique its own solutions does not improve performance. Typically, increasing compute at test time tends to yield better results. Why does this not hold in this case? Could the authors elaborate further on the intuition or mechanisms behind this observation?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The improvements over baseline models are substantial.\n* The paper is clearly written and easy to follow.\n* The experiments are carefully designed, with appropriate baselines and ablation studies.\n* Overall, the proposed approach is simple yet effective, and the empirical results are convincing.", "weaknesses": "* The idea of using critique data is not entirely novel, as similar concepts have been explored in supervised fine-tuning (SFT). However, to the best of my knowledge, this is the first work to apply critique data within an RL setting with verifiable rewards.\n* The contribution is somewhat incremental, as it mainly involves augmenting an existing RL framework with additional data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)