Critique-RL: Training Language Models For Critiquing Through Two-Stage Reinforcement Learning
Authors: Zhiheng Xi
Jixuan Huang
Xin Guo
Boyang Hong
Dingwen Yang
Xiaoran Fan
Shuo Li
Zehui Chen
Junjie Ye
Siyu Yuan
Zhengyin Du
Xuesong Yao
Yufei Xu
Jiecao Chen
Rui Zheng
Tao Gui
Qi Zhang
Xuanjing Huang
Venue: ICLR 2026 Poster (2026)
Keywords: large language model
Critique models
LLM reasoning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose Critique-RL, an online RL framework for developing critique models without stronger supervision, improving both the discriminability and helpfulness of critiques through a two-stage optimization strategy.

Abstract: Training critiquing language models to assess and provide feedback on model outputs is a promising way to improve LLMs for complex reasoning tasks. However, existing approaches typically rely on stronger supervisors for annotating critique data. To address this, we propose Critique-RL, an online RL approach for developing critiquing language models without stronger supervision. Our approach operates on a two-player paradigm: the actor generates a response, the critic provides feedback, and the actor refines the response accordingly. We first reveal that relying solely on indirect reward signals from the actor’s outputs for RL optimization often leads to unsatisfactory critics: while their helpfulness (i.e., providing constructive feedback) improves, the discriminability (i.e., determining whether a response is high-quality or not) remains poor, resulting in marginal performance gains. To overcome this, Critique-RL adopts a two-stage optimization strategy. In stage I, it reinforces the discriminability of the critic with direct rule-based reward signals; in stage II, it introduces indirect rewards based on actor refinement to improve the critic's helpfulness, while maintaining its discriminability via appropriate regularization. Extensive experiments across various tasks and models show that Critique-RL delivers substantial performance improvements. For example, it achieves a $9.02\%$ gain on in-domain tasks and a $5.70\%$ gain on out-of-domain tasks for Qwen2.5-7B, highlighting its potential.

Published as a conference paper at ICLR 2026
CRITIQUE-RL: TRAINING LANGUAGE MODELS FOR CRITIQUING THROUGH TWO-STAGE REINFORCEMENT LEARNING
X Jixuan Huang" Xin Guo', Boyang Hong', Dingwen Yang', Xiaoran Fan', Shuo Li", Zehui ,hen', Ye', Siyu Yuan', Zhengyin Du?, Jiecao Chen', Rui Zheng', Tao Guilt, Qi Zhang', Xuanjing Huang! Fudan University BytteDance zhxi 2m... fudan. edu. en, tgui fudan edu.
Zhiheng Xixxt,
ABSTRACT
Training critiquing language models i to assess and provide feedback on model out- puts is a promising way to improve L.L.s for complex reasoning tasks. However, existing approaches typically rely on stronger supervisors for annotating critique data. To address this, we propose Critique-RL, an online RL approach for develop- ing critiquing language models without stronger labeling. Our approach operates a two-player paradigm: the actor generates response, the critic provides feedback, and the actor refines the response accordingly. We first reveal that relying solely on indirect reward signals from the actor's outputs for RL optimization often leads to unsatisfactory critics: while their helpfulness (i.e., providing constructive feedback) improves, the discriminability (i.e.. determining whether a response is high-quality or not) remains poor, resulting marginal performance gains. To overcome this, Critique-RL RL. adopts two-stage optimization strategy. In stage I, it reinforces the discriminabilityy of the critic with direct rule-based reward signals; in stage II, it introduces indirect rewards based on actor refinement to improve the critic's helpfulness, while maintaining its discriminability via appropriate regularization. Extensive experiments across various tasks and models show that Critique- RL delivers substantial performance improvements. For example, it achieves a 9.02% gain in- mn-ommin tasks and a 5. 70% gain out-of-domain tasks for Qwen2.5-7B. highlighting its potential. 2
INTRODUCTION
With the development of large language mod- els (Ouyang et al. 2022 OpenAl 2023 Tou-
Owend 5-38 on MATH
Scaline Informee Cormate en MATI
13%
vron et al. 2023 Jing et al. 2023 Dubey et al.
2024), providing reliable supervision for them
has become critical research challenge (Bow-
man al. 2022, Saunders et al. 2022 Chen
(01)
et al.) 2025 especially for tasks that are diffi- cult even for humans, such as complex reasoning, sequential decision-making, and coding (Shinn al. 2023) Snell et al. 2024, Qu al. 2024 Kumar et al. 2024 Xi al. 2024a). This prob- lem is often referred to as scalable oversight (Bowman al. 2022). One effective method for scalable oversight is to train critiquing lan- guage models assess and provide feedback to
606
Duech 21 c300
Nache saaepies
Figure 1: Left: Critique-RI RL. achieves better per- formance and discrimination MATH. Right: Inference compute scaling for Critique-RL, with @ 2k and @3k indicating sampling amounts that are 2 times and 3 times the x-axis value, respec- tively. Critique-RL improves the performance ceil- ing and is more compute-efficient
'Equal contribution.
1 Corresponding author.
It can also be referred to as critique model or critic.
Our code are available

---

Published as a conference paper at ICLR 2026
Baxelin:: with Indirect Reward Signals
Problem: I hag of mortiss corn 530 and the picce incrusse by 2015 of orggial
pase evey two months how much would a bag murttes own ater moethor
Actor Model (original response):
C
Autor Model
@ reppoenn
Créique Model RL
Let' tress in donu by sey:
1. The priee of lag of markko ircnaso by 2011 001510 S4 evey bou eaaile.
Y criinge
2. monte, pocr the hey of mambie will have incresee 250phtpet trmes prize the bog
Revard Faxiion
incrosed
reporso @
reframmnii
Alllicy Aif i
56.
Verifier:
Currecsess Verifier
Purilu// / Feoenoctio
Critique Model (eritique)) momiis
20100r520-54 every
Oane Crilique-RL
Currectnes the stpp Correct
Explonatio:: to sap coroeity calcslap the incrave srice every two mentts, which 70% the original price of $T The cakodation 50 53n 54.
Stage I
Diserinizationn Verifier
4 Filli
Siep wattener
muntis, the price the log maablo will have incrased
repeese
ciliqzz
Reward Functom
Corrctton the Winnng
Explenation: Tai woong. The price lovrases every two moniss, the ouzber of inavases is mandiai to 16/22 I& / (2"2) -9. Thene are 18 twc-mantth periods in 16 momonhh
>>>
Actor Modal
Criigu Model eioa
Totaged
wifs
Correctocs fixal umswer: Weong
Diceituination Veriffer:
Actor Model (refinement response):
Lat's truac It doeec rap by sxye
Stuge II
Discrimination Verifier
Icsemiie the
beg otaaiiis
neal consder
resoese
ciiqqe
1. Initaa Friee: The ratial price
muthoa. Sin.
2. Price Incresse Calealation: The price ixveases by 20% of origaal price evey two moniis The incvas in price onay tws meraa iecakodaadi as $21
3. Namber Incrasanc prize trmuau many ton murtts, and thee monthe wis. centher iareeses 36norls 22
Actor Model
narpanee
Critiue Model
Rewad Fezeico
moniiis caleabana parted lotal incvase "
C criigee
$72.
mulhsphyngg the rumber
refisareni riquor
ARICK Tram I|I
uphiic will rovun)
S. Fixal Price Caloabttin: The find price of the bag markes aller eseetta is the intial price phe usaal inmasse Final price 572 592.
O Freffns
Cursetraow Varifier
Carrectasss Veriffer:
Figure 2: Left: A case illustrating the two-player actor-critic interaction, including the original response from the actor, the critique from the critic, and the refinement from the Actor. Right: Overview of our method and its comparison with baseline RI. The snowflake icon a on the Actor indicates that it is fixed, while the fire icon on the Critic indicates that it will be updated. Our method employs two-stage RL process. It optimize discriminability of critique models in Stage I, and optimize helpfulness while maintaining discriminability in Stage II.
model outputs Welleckk et al., 2023;
Akyürek
jet al. 2023 Xi al. [024bb Yao et al. 2024). Based on this feedback, actor models can refine and optimize their behavior outputs.
Existing work in training critique models typically assumes a stronger supervisor to provide labeled critique data, which is often expensive and difficult to scale (Saunders et al. 2022 Xi et al. 2024b Bowman et al. 2022). Moreover, the data labeled by the supervisor often differs significantly from the learner's output distribution (Kumar al. 2024). Another line of work does not train the model but instead relies on the model's inherent abilities, using prompt engineering to elicit its critiquing abilities (Bai al. 2022 Madaan al. 2023] Dhuliawala et al., 2024). However, such methods typically assume an oracle verifier during testing, allowing the critique model to bypass discrimination (i.c.. determining whether response is high-quality) and instead focus only on offering helpful feedback for revision (Xi et al. 2024b Gou et al. 2024). Without the oracle verifier, they often meet
performance bottleneck (Huang et al. 2024).
In this work, we aim to develop critiquing language models without relying on stronger labeling or an oracle reward function during testing. To this end, we propose Critique- RL, an online RL approach based on two-player actor-critic interaction (Yao et al. 2024 Xi et 2024b) for developing critique models. In our approach, there are two main roles: the actor and critic. The critic assesses (discriminability) and provides natural language feedback (helpfulness) for the actor's 's output, and the actor performs refinement accordingly (Saunders et al.) 2022).
To build our method, we first use the correctness of the actor's two attempts to shape the reward signals for the RL optimization of critique models ($..1). following approaches like Retroformer (Yao al. 2024) and CTRL (Xie et al.) 2025), where such indirect signals are shown to reflect the quality of critiques. However, this approach fails to develop satisfactory critique models, i.c. with low performance. Delving into the optimization process, we reveal that while the helpfulness of the
2

---

Published as a conference paper at ICLR 2026
critique models improves, their discriminability is not well optimized, leading to optimization bottleneck and even a collapse of RL training.
To address the challenges, Critique-RL RL employes two-stage RL approach ( 4.2 Specifically, as shown in Figure 2 in the first stage, we optimize the discriminability of the critique models using direct rule-based reward signals. In the second stage, introduce indirect rewards based on the correctness of actor refinement to enhance the helpfulness, while using appropriate regularization to maintain their discriminability. In-depth training dynamics shows that our method addresses the training collapse and stably optimizes both discriminability and helpfulness. Extensive experiments show that our method outperforms baselines across different models and tasks, yielding 9.02% improvement on in-domain tasks and 5.70% improvement on out-of-domain tasks for Qwen2.5-7B. also noteworthy that critique models trained with our method can generalize to unseen tasks, demonstrating its promise for scalable oversight.
In summary, our main contributions are:
Delving into the RL optimization process, we reveal that solely depending on indirect reward signals of actor's 's output correctness cannot develop effective critique models, which poses conflict and optimization challenges between the discriminative and feedback capabilities of crities.
2. We then propose Critique-Re RL, novel two-stage RL approach develop critique models for providing accurate assessment and helpful feedback for model outputs.
3. We perform in-depth experiments, ablation and analysis to show the effectiveness and stability of our method. We hope our work provides insights for the community.
2 RELATED WORK
Prompt engineering for eliciting critiquing ability from language models. As a key technique for scalable oversight (Bowman et al.] 2022), many previous works have explored the use of prompt engineering to elicit the critiquing and reflection abilities of LLMs (Bai et al.] [2022] Madaan et al. 2023 Ye et al.) 2023; Dhuliawala et al., 2024). These methods typically rely on an oracle verifier including answer matching or external tools at test time for discrimination. allowing the LLM to focus solely on providing natural language feedback (Xi et al.) 2024b] Huang et al., 2024). However, in the absence of external verifier, even SOTA models face significant challenges (Saunders et al.,
2022 Welleck et al.) 2023) Xu et al., 2024 Huang et al.) 2024). In this work, we do not assume an oracle verifier; instead, we train critique models through RL to optimize both discriminability and the ability to provide helpful feedback.
Fine-tuning language models for critiquing. Previously, a line of work has explored fine-tuning- based approaches for training critique models (Saunders et al. 2022 Bowman et al.) 2022 Xi et al.) 2024b). However, these methods primarily rely on a stronger supervisor for data annotation, which is costly and difficult scale (Xi et al. 2024b). To address this issue, some researchers have proposed self-improvement-beee bbaed methods to train models for self-critiquing (Tang et al. 2025 Zheng et al., 2024: Yuan et al.) 20256). Unlike these approaches, adopt two-player paradigm and train separated critique model through RL.
Reinforcement learning for language models. RL has become an essential component of LLM post-training, such RLHF for alignment (Ouyang al. 2022 Zheng al. 2023; Wang al.
2024; Shao et al.] 2024, Chen et al. 2024). Additionally, various works have leveraged RL to reasoning (Snell et al. 2024| Kumar et al.) 2024 Xi et al. 2024a). coding (Kumar et al. 2024). and decision-making tasks (Shinn et al. 2023 Xi et al. 2025b). Furthermore, some studies explore using RI to improve LM's ability for relf-reflection and self self-correction (McAleese et al. 2024 Kumar et al.) 2024 Welleck et al. 2023; Shiin et al.) 2023 Xu et al., 2024] Ye et al. 20233 Yuan et al. 2025a). Other methods, such as Retroformer Yaa et al. 2024) and CTRL (Xie et al. 2025), leverage indirect reward signals to optimize critique model's 's helpfulness, targeting decision maaiing tasks and coding tasks, respectively. However, their RL phase overlooks the joint optimization of discriminability and helpfulness. Different from them, we propose a two-stage Critique-Ro RL approach to optimize both discriminability and helpfulness, effectively developing critique models.
enhance language models' performance
3

---

Published as a conference paper at ICLR 2026
3 PRELIMINARIES
3.1 THE TWO-PLAYER INTERACTION FRAMEWORK
The multi-agent framework in this work consists of two main roles (Yao et al. 2024 Xi al. 2024b): the actor model and the critique model. It operates through a
process.
Specifically, given question 2, the actor model is expected generate original response y To(x), which includes both the reasoning trajectory and the final answer. The correctness verifier then provides oracle reward Forele (3r, y) to the actor model. Subsequently, the critique model T takes the question-respons. pair (I. as input and produces critique [z y), which should include assessment the response correctness (discriminability) and offer constructive natural language feedback (helpfulness). Based on this critique, the actor model generates a refinement response y [z y, c). and subsequently receives an oracle reward Foracte (2, y Using these rewards, ie.. Foracle (z. y) and Torelee (2, y'), we can design different reward functions r-(+) for critique models,
which will be shown in $4
3.2 POLICY GRADIENT FOR LLMS
Policy gradient methods (Sutton et al. 1999 e.g., REINFORCE (Ahmadian et al.
2024 Kuman
et al. 2024), are common techniques to perform RI on LLMs (Ouyang et al. [022] Xi et al. 2025a).
For the policy critique model & parameterized by é, the objective of policy gradient is to find an optimal policy that maximizes the reward function r-(-). ). It is typically expressed as maximizing:
E Bowxx((12 w..
[re y, c, )],
where (e aoo'rrrrl (Z,g,)) denotes the expectation over the critique sampled from the critic and the refinement response sampled from the actor тæ. This gradient is used to optimize the critique model via gradient ascent. The positive critique "reinforced" by increasing its probability.
3.3 EVALUATION METRICS
To evaluate the performance of the critique model, we consider the following metries: (1) Acc@ Refine: the accuracy of the actor model's refinement response; (2) A: the improvement in the actor model's accuracy between the original and refinement response, which measures the effectiveness of the critique model; (3) A c-ti; the change rate from an originally correct response to incorrect refinement response. A lower value is better; (4) A i++c; the change rate from an originally incorrect response to correct refinement response. A higher value is better; (5) Ace@ Dis: direct metric to measure the discriminability of the critique model, which quantifies the accuracy of whether the correctness accessed by the critic aligns with the true correctness of the original response.
METHODOLOGY
4.1 MOTIVATING FINDINGS: RL WITH INDIRECT REWARD SIGNALS IS INSUFFICIENT FOR TRAINING SATISFACTORY CRITIQUE MODELS
In the two-player actor-critic framework (aao et al. 2024 Xi et al. 2024b), a natural and intuitive way optimize the critiquing language models is to shape the reward signals derived from the actor's two attempts (original and refinement responses). We explore several reward shaping approaches, demonstrate their failure modes, and investigate why they fail to incentivize satisfactory critiquing ability.
Analysis setups: data, models, and training methods. Our preliminary experiments are GSM8K (Cobbe et al.] (202T), and the backbone model is Qwen2.5-3B (Team] 2024). Following previous work (Xi al.) (024b), we train an actor model capable of generating responses and reasonably following critiques. To build the SFT dataset for initializing a base critique model, prompt Qwen2.5-3B Siven..-33-Instrucct Instruct to obtain critique data DsFt {x. y, c) c]
Paarl rather than using
annotations from SOTA commercial models like GPT-4o (OpenAl 2023). We filter the critique data based on the correctness of refinement to ensure the quality.
4

---

Published as a conference paper at ICLR 2026
RoegRefine
4'
4
80
2.-
74
24
2.-
78
77-
Training fraction
Training fraction
Training fraction
Ace@Dis
AccoiDis Originally Correct
Ao Originally Incorvect
70
85.0
90.9
00.0
87.5
Training fraciion
Training fraction
Toxming fractoo
Baseline(Fretme)
Raseline(Fomcccin)
Baseline((a)
Critique 1tape
Crtique RL Stage
Figure Training dynamics preliminary experiments. "Acc@ Dis Originally Correct" and "Ace @ Dis Originally Incorrect" refer to the discrimination accuracy of originally correct and incorrect responses, respectively. Baselines using indirect reward signals to optimize helpfulness tend to exhibit overly conservative aggressive behavior the discriminability our Critique-RI. optimizes discriminability in Stage I, and optimizes helpfulness while maintaining discriminability in Stage II, achieving better in Acc @ Refine, A c- and A i
well optimized. In contrast,
Next, we train critique model Tτ using the SFT loss:
CSFT (6) B(z.y.))
log TT (clxx)
(2)
We then employ policy gradient (Sutton et al. 1999) to maximize:
BL Bownly ma))a m.see) ro(2, y, C, - (KL SFT (cla, 1) (cla, v))
(3)
where mo the fixed actor model, SFT the SFT model. Each x is a query sampled from the dataset DRL y is the original response. KL(-II-) means the KL--divegeecc which constrains the distance between the RL model and the SFT model, and B is scaling factor. rz(-) is the reward function for critique models. Here, with r Foracle being the oracle reward function that verifies the correctness of an actor response, r-(-) can be Trefine which represents the correctness of the refinement:
Trefine (v, y, yy) = Toracle (I, v/),
or it can be rA which represents the difference in correctness between the actor's two attempts:
TA (x. B, c, = Foructe (2, y/ - Turacte (z. y)-
(5)
Moreover, also include I reoorection re(-) for reinforcing the ability correct incorrect responses:
1.0, roracte (x, y) = and Foracte (x. s/) =
Teorrction (2. y, (,,"),
0.2, r ooract (x, y) = and Toracte (z, v/) 1,
0.0, Toracte (z, y/) = 0.
Empirical findings and behavior analysis. We illustrate the training dynamics during RL in Figure Optimizing with Trefine and can reduce A c-v preventing originally correct responses from being altered incorrectly, tion performance is not good enough. This phenomenon reveals that the critique model is overly conservative, encouraging the actor to not change answers. As a result, the final Acc Refine not satisfactory.
its A i-+c is not significantly optimized, meaning its error correc-
In contrast, optimizing with Foorrccton improves +o but fails to effectively reduce A c-+ This means it often provides more aggressive suggestions, encouraging the actor model to correct incorrect responses, but it also introduces a greater risk of turning originally correct answers into incorrect ones. Similarly, the final Acc @ Refine is also not satisfactory.
5

---

Published as a conference paper at ICLR 2026
Algorithm 1: Critique-RI
Input: Actor model To. base critique model TT. SFT dataset PSFT. RL dataset DRI. function that extracts the correctness of response judged by a critique J. oracle reward function Toracle discrimination reward function rdis.
Procedure Supervised Fine-tuning:
SFT
TTi
Update T by minimizing (oF B(r.a.e)-Psr log Tog w)}:
Procedure Critique-RL Stage I: optimizing discriminability through direct reward signals.
Stage =PT:
for batch in DR. do
for x in batch do
Stage-1,
Generate and with my and n
Compute discrimination reward with rais (r, y, ()) = -1(0)),1(,,)))))) v. Toracle (2, v)):
end Stage-1 T by maximizing
Update
E sag (+wa,)
(a, y) (K (F (c/2, =m Stage (clar, v))
end
Procedure Critique-RI Stage II: optimization helpfulness while maintaining discriminability.
Stage.II II
Stage
+
for batch in Dri. do
for æ in batch do
Generate y, e cadd y with mg and Stago=1l;
Compute discrimination reward with (a = -1(((x, Toracte (2,, v)):
Compute refinement reward with Pretine (2 1/):)
end
Update
Stage-1l &
by maximizing E
Bewn (lag))o Prefine Birais(E,, y,,
32KL Stagel (cl v)) a
Stage-1l Steee (clx,y))
end
Analyzing underlying reasons for the failure modes. To reveal the reasons behind the above failure modes, also visualize the discrimination performance of the critiquing language models during RL in Figure 3 We find that as RL progresses, all three reward functions Trefines and Peonection fail optimize discriminability effectiivlyy. For originally correct and incorrect responses, they can only optimize the judgment for one, while the ability to judge the other is reduced. This may because both of the indirect reward functions are based on the actor's responses, targeting helpfulness and overlooking discriminability. This motivates the proposal of our method.
4.2 TWO-STAGE CRITIQUE-RI.
Key challenges. Based on the previous analysis, have identified key challenges in RL for critiquing language models: (1) optimizing the discriminability of critique models to improve their accuracy in judging both correct and incorrect original responses; (2) improving the quality of the model's 's feedback, i.e., helpfulness, while maintaining its discriminability, to prevent the issues of being overly aggressive or overly conservative.
Method overview. To address the above challenges, we propose the two-stage Critique-RE RL. In the first stage, our method explicitly optimizes the discriminability of the critique model using direct reward signals. We then the resulting model o the second stage, introduce reward function based on the actor's response optimize the critic's
Stage-1 as the initialization for the second stage. In

---

Published as a conference paper at ICLR 2026
helpfulness, while also incorporating appropriate regularization to maintain its discriminability. We illustrate our method in Figure 2 and the algorithm is summarized in Algorithm I
Stage I: optimizing discriminability through direct reward signals. We decouple the discrim-
inability and helpfulness of the critique models (Saunders et al. 2022 Chen al. 2024). In Stage I, we shape the reward based solely on the actor's ' original response. Given (z. critique models prompted give correctness judgments for each step, and also provide a judgment for the final answer. Based on this, we define the discriminability reward function of the critique models as:
Tais (2, y, c) = (J(Z. c) = Taradle (Trade((r,1)) (u,u)):
where I (ar W/ e) is the critique model's judgment of the correctness of the original response. 1() is indicator function that returns only when the condition inside the parentheses holds, and 0 otherwise. Based on this, our Stage I RL maximizes:
E
Stwel (- (-1a,y) I'ii (z, v)c) (KL( ( 1) Stuge
(8)
Come
where the KI. divergence with the SFT model still used to stabilize the training. As shown in Figure 3] our Stage I RL can effectively and stably optimize discriminability, regardless of the correctness of the original response.
Stage II: optimizing helpfulness while maintaining discriminability. The goal of the second stage of Critique-RL discriminability. thereby avoiding overly conservative or overly aggressive behavior patterns. To achieve this, we introduce a reward function "reline based on actor refinement correctness. Meanwhile, to preserve the model's ' discriminabilityy we retain rais and introduce a regularization term based on maximize the following objective:
optimize the helpfulness of the critique models without sacrificing their
Stage-1 Specifically, the KL divergence with the Stage I model T
Com E Sage-ll (-12.)). wowll Prefine + 3irais(z, y, c) 32 Stage 1 y) IT 1
Stuge (c|Ix n))
where BR and 32 are scaling factors. As shown in Figure our Stage II effectively optimizes the model's helpfulness, increasing A iio and decreasing c->i ultimately leading a stable improvement in Acc@ Refine and A, Our method also performs strongly on the test set (see $5)
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Focusing on mathematical reasoning tasks, we select different commonly-used tasks, including free-from and multiple-choice. Following Ding et al. (2025), we construct training set with the train-split of MATH (Hendrycks et al.) 2021). GSM8K (Cobbe et al. 202I), AQUA (Ling et al. 2017). The testset of the three tasks are used as in-domain testset, while the test-split of SVAMP
Datasets.
Patel et al. 2021), TheoremQA (Chen et al.) 2023). are used as our OOD (out-of-domain) testset.
Models and baselines. Our experiments are mainly conducted on Qwen2.5 series (Team 2024). i.., Qwen2.5-3B and Qwen2.5-7B. Besides, we also conduct experiments other models like Qwen2.5-72B, I.lama.22 (Dubey et al. (024)) and DeepSeek-R -neeee-RR-Distti---weee
(DeepSeek-AAl
2025) (see Appendix C and Section 6). We include several baselines: (1) SFT which fine-tunes models with critique data. (2) STaR (Zelikman et al., (022)) which iteratively fine-tunes critique
models on self-generated data and filtered based on the refinement correctness of the actor. (3) RL
baselines that leverages indirect outcome- based reward as baselines, i.e., Retroformer (aoo et al.
2024 which uses PPO and CTRL (Xie et al. 2025) which uses GRPO.
Implementation details. All experiments are conducted on NVIDIA A800 GPUs. To initialize an actor that can reason and refine based on the critiquing feedback, we follow Ding et al. (2025); Xi] ct al. (202bb) to construct dataset of 21, 973 reasoning traces and 12, 000 refinement responses. For critique data, we construct set of 6, 000 examples, with 2, 000 examples in each training task. For tine-tuning actors, we set epoch to and learning rate to 5e - 6, and remains fixed during further training phase; for fine-tuning critics, set epoch to 5 and learning rate to 5e 6. We use the same base model for the actor and the critique model. For STaR and RL, we perform SFT to obtain an initialized model. In RL, set KI. coefficient to 0.01. In Critique-RI, use RI.O base
7

---

Published as a conference paper at ICLR 2026
Table 1: Main results. The best performance is in bold and underlined, while the second-best performance underlined. Our method marked in blue No Critic means the actor model perform reasoning only, and we report the reasoning performance. For other methods, we report the Acc@ Refine performance for the acc column.
MATH
GSM8K
AQuA
Model
Method
Acc
Acc@Dis
Acc
Acc Dis
Acc
A
A
No Critic
36.90
66.03
50.00
SFT
44.24
7.34
66.51
69.14
3.11
76.34
46.46
-3.54
61.97
STaR
44.38
7.48
66.97
71.95
5.91
74.79
50.39
0.39
66.13
Qwen2.5-3B
Retroformer
44.54
7.64
65.11
70.51
4.47
77.59
51.18
1.18
58.44
46.14
69.29
70.58
76.71
53.54
3.54
62.20
Critique-RL
48.60
11.70
82
75.89
9.86
87.44
56
6.69
69.92
No Critic
45.74
75.66
63.39
SIT
51.84
6.10
67.50
78.77
3.11
79.42
59.45
3.94
68.67
Qwen2.5-7B
STaR
54.06
8.32
69.71
80.52
4.85
81.03
57.87
5.51
72.18
Retroformer
52.34
6,60
68.03
80.82
77.05
03.39
0.00
70.56
CTRL 53.86
8.12
71.42
81.35
5.69
83.44
64.96
1.57
71.66
Critique-RL RL
58.40
12.66
85 20
87.72
12.05
90.43
65. 75
2.36
78.09
algorithm performs well and does not require a value model. In Stage II, is set to 0.2. We train the critique model for 500 steps at each stage and report best results. During evaluation, the temperature set to For inference-compute scaling and Pass@ K, we set temperature to 0.7.
5.2 MAIN RESULTS
Generally, critique models can significantly improve actor's reasoning performance. The re- sults in Table demonstrate that when introducing critique models, the actor's reasoning performance can be boosted by a large margin. For example, the MATH task, even the SFT Baseline outperforms the model without critic by 7.34 and 6.10 points on the 3B and 7B models, respectively. This suggests that critique models are an effective scalable oversight method, as discussed in Saunders et al. )022);; McAleese et al. 2024).
RL-based methods outperforms mme-tuning--asee ones. Both SFT and STaR methods lead to promising critique models, but in most cases, online RL-based methods perform better, especially our Critique-Re RL. For instance, on the 3B model, our method surpasses the SFT method by an average of 7.11 points accuracy across three datasets. It is worth noting that AQuA, fine-tuning- based SFT and STaR may lead to negative impact on performance, while our method provides significant positive improvements. This reveals that online RL methods have greater potential and adaptability in eliciting the model's i critiquing ability, similar to the findings in McAleese et al. (024).
Critique-RL RL consistently outperforms other baselines in discrimination and final accuracy. In terms discrimination, our method also significantly outperforms other baselines, such as surpassing CTRL by 5.31, 6.36 points for 3B and 7B models GSM8K, respectively. This reveals that discriminatiion-rlated reward shaping can effectively optimizes discriminability. Thanks to this and the helpfulness reward design in the second stage, method shows significant improvement in final performance compared to other baselines. For example, on the 7B model, our method outperforms Retroformer by average of 5.11 and 12.69 points accuracy and discriminability, across three datasets.
5.3 ITERATIVE IMPROVEMENT OF CRITIQUE-RI RL
Iterivve Refinement: MATII
Ieerative Refinemet: GSMRK
Furthermore, we validate the iterative improve- ment capability of Critique-RI RL through two key aspects: (1) Iterative refinement process: Dur- ing the i-th iteration, the critic generates cri- tique e; (" yo, C1, 1, 3i-1), while the actor produces the refined response y = (1 yo. C1, (;) accordingly. (2) Itera- tive training process: We alternately conduct the two-stage training of Critique-RI RL (Stage I and
Number samples
Number samples
50 ertic
Chique-A
Figure 4: Results of iterative critique-refinement of Critique-RL using Qwen2.5-3B.
8

---

Published as a conference paper at ICLR 2026
Stage II) to optimize the critique model. The detailed results are shown in Figure and Table 2 respectively.
First, as demonstrated in Figure 4 through itera- tive critique and refinement, the model exhibits consistent Acc gains Qwen2.5-3B, with each iteration achieving measurable improvements. Second, iterative training leads further perfor- mance enhancement, with detailed results using Qwen2.5-3B on MATH dataset shown in Ta- ble 2 Specifically, both Stage I and Stage II of Critique-RI. demonstrate consistent improve- ment in Acc and Acc@ @ Dis metrics. Compared to the first iteration, the second iteration improves by 2.40 and 3.68 points on accuracy and discriminability.
Table 2: Results of iterative training of Critique-Ri RL using Qwen2.5-3B on MATH.
Method
Acc
Āccie Dis
No Chilie
36,9
SFT
44.2
7.3
66.5
Iteration Stage
45.9
9.0
75.7
Iteration I. Stage II
48.6
11.7
82.8
Critique-RL
Iteration 2. Stage
49.5
12.6
85.0
Iteration 2. Stage
14.1
86.5
DISCUSSION AND ANALYSIS
Ablation on different stages. We con- duct ablation experiments to validate the importance of different components. The results are shown in Table Both Stage I and Stage II are crucial, and removing either of them leads to performance drop. This indicates that optimizing both discriminability and helpfulness is essen- tial in developing critique models.
Table Ablation study using Qwen2.5-3B. We report the Acc @ Refine. "w/o" means without; "Stage II w/o discrimination" means in Stage II, we remove Tdis and KL Stage ||
"Stage II ww" and "Stage II w/ Stuge |I
Feoeection mean replacing the Treine with the corresponding reward function.
Method
MATH
AQUA
Ace @ Refine
&
Aec@Reffnn Refine
Acc Dis
Crnique-R (Ours)
Fi
82.1
Ablation reward design for Stage II Next, we perform a deeper anal- ysis of the reward design in Stage II. First, if we remove the discrimination- related rais and KL-based regularization Stage- ( ||TT || ). the discriminability and accuracy suffer a significant drop. This further empha- sizes that when optimizing for helpfulness, is crucial to maintain the model's discrimination ability. Second, when we replace the reward function Trefine in Stage II with another reward function, i.e., and coorecction we observe slight performance drop. This may be because Trefine directly optimizes the Acc @ Refine metric, which aligns most closely with the test-time scenario.
-wlo Stage I
17.6
79.7
53.9
66.5
-wl Stage II
45.9
78.7
54.7
68.22
-Singe wdo serimimatiinn
77.7
53.5
61.6
-Saage w/ ra
48.2
82.6
53.9
08.4
-iinge wi Yermecion
47.7
82.0
54.7
68-4
Singe=")),
Analysis of helpfulness when the oracle verifier is available. Many previous works have relied on an external oracle verifier assess the actor's reasoning results (Bai et al. 2022 Madaan et al., 2023 Ye et al. 2023 Dhuliawala et al.) (024). In this scenario, the model's judgment ability is isolated, allowing us to better evaluate the critique model's helpfulness. We conduct relevant experiments. and the results are shown in Figure We find that when the oracle verifier is available, all baselines show performance improvements. In this case, our method still outperforms others across different datasets and models, indicating that our approach significantly enhances the model's helpfulness. Furthermore, comparisons with other RL baselines reveal that the optimization of discriminability in method also implicitly contributes to the improvement of helpfulness, suggesting that the two abilities are not entirely independent. This further emphasizes the importance of optimizing both abilities jointly in developing critique models.
Evaluation of test-time inference compute scaling for Critique-RL. We investigate whether Critique-RI. can be combined with inference-timet compute scaling strategy. Following Qu al.) (2024); Snell et al. (2024): Xi al. (2024b), we leverage the commonly used majority vote (MV @ K) (Wang et al. 2023) which evaluates whether the most frequent answer among K samples is correct. The results of MATH are shown in Figure I and the results of GSM8K shown in Figure 6 of Appendix F Compared to the bascline, Critique-RL. significantly increases the performance ceiling and shows more sustained upward trend as inference compute scales. More importantly, performing K x response-erittquu-reineee reetieettteeeeteeeeetttt sampling is more compute-efficien efficcien than conducting 3 x parallel sampling responses, suggesting the compute--fficiency of Critique-RL.

---

Published as a conference paper at ICLR 2026
Qwee2.5-3B B on MATH
Ques2. 2-3 en GSMBK
Qwar2.5-B5 5-38 on AQuA
cradle vantker
123
70
73
46
Chiige
Chlige
Qwea2.5-7B
Qwes2,5-78 GSM8K
Qwea2.5-778 AnAA
21+
63
711
C19
344
19
STA8
CTRL
Chiipee RD
SFT
838
CTRL
Chiipe RL
CTRL
CVilien
Figure 5: Performance with and without the oracle verifier. When the oracle verifier is available, the model longer needs make discrimination and just needs provides useful feedback. This allows us to evaluate the model's 's helpfulness more accurately.
Generalization to OOD tasks. We also validate the generalization of the models trained by Critique-RL on OOD tasks. The results in Table 4 show that the models trained still delivers significant performance improvements, further demonstrating the po- tential of this scalable oversight approach.
Table 4: Out-of-domain evaluation of Critique-RA RL
SVAMP
TheoremQA
Model
Method
Pas
Crtii
70.7
92.0
15.1
318
SFT
74.7
95.7
15.3
36.1
Qwen2.5-3B
Retroformer
75.0
96.0
16.1
37.0
CTRL
76.0
95.7
15.8
36.5
Critiquc-RL
78.3
16.8
37.8
No Critic
80.3
95.7
19.4
39.8
SFT
83.0
95.7
20.5
11.9
More experiments and qualitative analy- sis. We conduct extensive experiments to
Qwen2.5-7B
Retroformer
84.0
96.0
20.0
42.3
CTRL
85.1
96.7
21.1
42.9
show the effectiveness and working mech- anism of Critique-RL, with the detailed re- sults presented in the Appendix: (1) In addition to the Qwen2.5 series (Team 2024), we evaluate our
CritiqueRRT.
89.7
97.0
21.4
43.0
method on different architectures including Llama3.2 (see Appendix ( (2) We compare Critique-RL.
with other refinement methods including Self-Refine (Madaan et al.] 2023). SuperCorrect (Yang et al.)
2024) and Critic-Cot (Zheng al. (024)) and the results are presented in Appendix E
(3) We also
perform test-time scaling analysis of sampling multipe refinement on the same response. with results presented Appendix F (4) We conduct experiments on summarization tasks using CNN/DailyMann (Hermann et al. (013) dataset to investigate our method's 's generalization ability on open-ended tasks where rule-based verifier cannot be directly applied, the results are in Appendix G (5) We perform a
qualitative analysis on how Critique-RI RI. works and provide several examples in Appendix
7 CONCLUSION
In this paper, we propose Critique-RL, an RL approach for developing critique models. Through in-depth analysis, we highlight the importance of explicitly optimizing model discriminability and propose two-stage RI. approach that effectively optimizes both discriminability and helpfulness. We validate its stability and superiority through detailed experiments, and further uncover its working mechanism through ablation studies and analyses. We hope that our work can provide insights for the scalable oversight community of language models.
ETHICS STATEMENT
This paper presents Critique-RL, a novel two-stage RL approach to develop critiquing language models for providing accurate assessment and helpful feedback for model outputs. We firmly state that this work is intended for ethical and constructive purpose. While no immediate societal harms evident, this approach enables scalable supervision by training models with minimal direct human oversight. Nevertheless, its potential susceptibility to misuse warrants proactive measures to ensure responsible governance.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We claim our detailed experiment setting in S.1 In addition, we upload anonymized versions of our data and code in a Zip file with a Readme file to ensure easy reproduction of all reported results.
ACKNOWLEDGMENTS
The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by Henan Province Major Industrial "Challenge-Based Innovation" (No. 251000210300, National Natural Science Foundation of China (N0.62476061, 62376061, 62576106), Shanghai Rising-Star Program (23QA1400200), and Natural Science Foundation of Shanghai (232R1403500).
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back basics: Revisiting reinforce-style optimization for learning from human feedback in llm.. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 12248-12267. Association for Computational Linguistics, 2024, doi: 10.1665511220444CL-
URL
AIME. Aime problems and solution, 2025. URL wwwLiattsss//atttooooeeeeesssss
wi trlki/inde htt:://neex..pp//A/M Problems aadds_andd__ Solutions
Afra Feyza Akyürek, Ekin Akyürek, Ashwin Kalyan, Peter Clark, Derry Tanti Wijaya, and Niket Tandon. RL4F: generating natural language feedback with reinforcement learning for repairing model outputs. Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL. 2023, Toronto, Canada, July 9-14, 2023, pp. 7716-7733, Association for Computational Linguisties, 2023. doi: 20.18653/V1/202.0 ACW-LONGAAL
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08077 2022.
Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamile Lukosiute, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran- Johnson, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish Joshua Landau, Kamal Ndousse, Liane Lovitt, Nelson Elhage, Nicholas Schiefer, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Robin Larson, Sam McCandlish, Sandipan Kundu, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Timothy Telleen-Lawton Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Satfield-Dodds Ben Mann, and Jared Kaplan. Measuring progress on scalable oversight for large language models. CoRR, abs/22 11.021100340 2022. doi: 10.48550/ARXIV. 1.35400440
Lu Chen, Rui Zheng, Binghai Wang, Senjie Jin, Caishuang Huang, Junjie Ye, Zhihao Zhang, Yuhao Zhou, Zhiheng Xi, Tao Gui, Qi Zhang, and Xuanjing Huang. Improving discriminative capability of reward models in RLHF using contrastive learning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pp. 15270-15283. Association for Computational Linguistics, 2024. doi: 22.18653/V11/222..550 EEMNL-MMINE URL httphhtpp////tt wwaddoi.orgg/// 1ee.18653310 /224 .mm p ma 852
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang. and Tony Xia. Theoremqa: A theorem-driver question answering dataset. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural I Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 7889-7901
11

---

Published as a conference paper at ICLR 2026
Association for Computational Linguistics, 2023. doi: http:///111223..MMLMMPMMM URL httttsts///oiii.. wtt.:///....11.///10000 http:/2222.eem11333..
Wenxiang Chen, Wei He, Zhiheng Xi. Honglin Guo, Boyang Hong, Jiazheng Zhang, Rui Zheng, Nijun Li, Tao Gui, Yun Li, Qi Zhang, and Xuanjing Huang. Better process supervision with bi-directional rewarding signals. CoRR, abs/2503.04618. 2025. doi: (0.48550/ARXIV255000 04618.
URL htt.h:ttp///oootoo .org 48.5505044.00 aarpvarrvvvvvvvva//vvv .tl.v.2555533 2503. 04618
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110. 2021.
DeepSeck-AL. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URLp::::://arxiii wttp://www.rggab//55..11.1 12948
Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of -f-veiiiiiititiio reduces hallucination large language models. In Lun-Wet Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACI. 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 3563-3578. Association for Computational Linguistics, 2024. doi: 10.1110.16600
Yiwen Ding, Zhiheng Xi, Wei He, Lizhuoyuan Lizhuoyuan, Yitao Zhai, Shi Xiaowei, Xunliang Cai, Tao Gui, Qi Zhang, and Xuanjing Huang. Mitigating tail narrowing in LLM self relf-improvement via socratic-guided sampling. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lin- Human Language Technologies ( (Volume 1: Long Papers), pp. 10627-10646, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-891766 189-666-1166..
guistics:
URL https:tt//s/e http:/acllaaaa anthoo wwg.y.org/20000 202. naac loreecclllooge long. 533/
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aoho Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon. Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliaan Zarov, Imanol Arricta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. CoRR, abs/2407.21788, 2024.
doi: 0..48550/ARXIV.2400
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. CRITIC: large language models self-correct with tool-interactivv critiquing. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1. NeurlPS Datasets and Benchmarks 2021. December 2021, virtual, 2021. URL
12

---

Published as a conference paper at ICLR 2026
Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Gareett (eds.), Advances in Neural Information Processing Systems 28: Annual Conference on Neural In- formation Processing Systems 2015, December 7-12, 2015, Montreal, Quehec, Canada, pp. 1693-1701, 2015, URL httpp::/pproceedinnng. wttp:/oceeednnnnnnuiiii.ss....ss
wwtperippppcpppppp.peeeaaama htcppapeer/21///aaeeeeeeeeeeee 4fdec7005cc9F143022 www4 Efcer7700ccF111522200EEEEEEEEE33 Abstracttt.ttttttttttt.tt.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.068255, 2023. doi:
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M. Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal M. P. Behbahani, and Aleksandra Faust. Training language models to self-correct via reinforcement learning. CoRR, abs/2409. 12917, 2024. doi: 10.48550/ARRKA. ASS50/ARXXV.2409.1222 12117.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale gen- cration: Learning to solve and explain algebraic word problems. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pp. 158-167. Association for Computational Linguisties, 2017. doi: 10. 10.18655/V1/117-1015 URL
Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, Yang Liu, and Yahui Zhou. Skywork-reward-v2. Scaling preference data curation via human-ai synergy. CoRR, abs/2507.01352, 2025. doi: 10.48550/RXIV.2507. 01352. URL https: /oii /0t.:///...//00/15/////// aww.v. 2507. 01352
48550/
Xueguang Ma. Qian Liu, Dongfu Jiang, Ge Zhang. Zejun Ma, and Wenhu Chen. General-reasoner: Advancing Ilm reasoning across all domains. 14X2522505.14652, 2025. URL htttts://axxiii
org/abs/2505. 14652]
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye. Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.
Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. Llm critics help catch 1lm bugs. arXiv preprint arXiv:2407.002/5. 2024.
OpenAl. GPT-4 technical report. CoRR, abs/2303 2023. doi: 10.48550/ARXII.2330.07777
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L.. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Anmual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022.
13

---

Published as a conference paper at ICLR 2026
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? ? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.). Proceed- ings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLLT 2021, Online, June 6-11, 2021, pp. 2080- 2094. Association for Computational Linguistics, 2021. 168. URL URL httphhttp:///oom http://///oooooro/..0//////////0 naacl naacl main. 168
Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. CoRR, abs/2407. 2024. doi: 10.48550/ARXIV,
2407.18219.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URL http:::
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self- Self-critiquing models for assisting human evaluators. CoRR, abs/2206.05802, 2022. doi: 10.44550/ARXII.2200.055
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingehuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024. doi: 10.48550ARXXV.2400..05 https://doi .rg. /0.g/10.ee /t////10..850500 48550/555/20 2402. 03300
URL
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflex- ion: language agents with verbal reinforcement learning- In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Charlic Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. CoRR, abs/2408.0333 2024. doi: 10.48550/ CRRXIV2408.03314
Richard S. Sutton, David A. McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Sara A. Solla, Todd K. Leen, and Klaus-Robert Müller (eds.), Advances in Neural Information Processing Systems 12, [NIPS Conference, Denver, Colorado, USA, November 29 December 4, 1999], pp. 1057-1063. The MIT Press, 1999.
Zhengyang Tang, Ziniu Li, Zhenyang Xiao, Tian Ding, Ruoyu Sun, Benyou Wang, Dayiheng Liu, Fei Huang, Tianyu Liu, Bowen Yu, al. Enabling scalable oversight via self-evolving critic. arXiv preprint arXiv:2501.05727, 2025.
Qwen Team. Qwen2.5: A party of foundation models, September 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer,, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jereemy Fu, Wenyin Fu, Brian Fuller. Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fin-ttmned chat models. CoRR, abs/2307.09288 2023. doi: 00.4855/ARXXI..3377..
14

---

Published as a conference paper at ICLR 2026
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan. Zhiheng Xi, Jun Zhao, Xiao Wang. Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. Secrets of RLHF in large language models part II: reward modeling. CoRR, abs/2401 006080, 2024. doi: 248550/ARXX.22401..
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to celfecorrect. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, Honglin Guo, Wei Shen, Xiaoran Fan, Yuhao Zhou, Shihan Dou, Xiao Wang, Xinbo Zhang, Peng Sun, Tao Gui, Qi Zhang, and Xuanjing Huang. Training large language models for reasoning through reverse curriculum reinforcement learning. CoRR. abs/2402.05808, 2024a. doi: 10.48550/ARXIV.240222
URL https: /doi. org/10...//. 48550/arXiv
2402. 05808
Zhiheng Xi, Dingwen Yang. Jixuan Huang, Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong. Shihan Dou, Wenyu Zham, Xiao Wang. Rui Zheng. Tao Ji, Xiaowei Shi, Yitao Zhai, Rongxiang Weng, Jingang Wang, Xunliang Cai, Tao Gui, Zuxuan Wu, Qi Zhang, Xipeng Qiu, Xuanjing Huang, and Yu-Gang Jiang. Enhancing LLM reasoning via critique models with test-time and training-time supervision. CoRR, abs/241 166799, 2024b. doi: 1044550ARXXV.2411.6677 URL https:///oot http:////oooo0111.100303 48550100.5555255411
Zhiheng Xi, Xin Guo, Yang Nan, Enyu Zhou, Junrui Shen, Wenxiang Chen, Jiaqi Liu, Jixuan Huang, Zhihao Zhang, Honglin Guo, Xun Deng, Zhikai Lei, Miao Zheng, Guoteng Wang, Shuo Zhang, Peng Sun, Rui Zheng, Hang Yan, Tao Gui, Qi Zhang, and Xuanjing Huang. BAPO: stabilizing -oolpo reinforcement learning for lims via balanced policy optimization with adaptive clipping. CoRR, abs/2510.189211 2025a. doi: 10.48550/ARXI22510..9999
URL https
//doi. 48550 //000..or///0045333////222.21111 2510.
Zhiheng Xi, Jixuan Huang, Chenyang Liao, Baodai Huang. Honglin Guo, Jiaqi Liu, Rui Zheng, Junjie Ye, Jiazheng Zhang, Wenxiang Chen, Wei He, Yiwen Ding, Guanyu Li, Zehui Chen, Zhengyin Du, Xuesong Yao, Yufei Xu, Jiecao Chen, Tao Gui, Zuxuan Wu, Qi Zhang, Xuanjing Huang, and Yu- Gang Jiang. Agentgym-rl: Training LLM agents for long-horizon decision making through multi- turn reinforcement learning. CoRR, aww/aa/0s.0000/// 08755, 2025b. doi: 10. 10448550/RRXW.25000000 08755.
URL httpt:
org/10. 48550/ http:aaiiivv........ 2 2509. 08755
Zhihui Xie, Jie Chen, Liyu Chen, Weichao Mao, Jingjing Xu, and Lingpeng Kong. Teaching language models to critique via reinforcement learning. CoRR, abs/25002.03492. 2025. doi: 10.48550/ 03492
ARXIV.2502.03492 JRI.https: //dttps:://o..... /ooi. wrw//0....... 18550//aXiiv
Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Wang. Pride and prejudice: LLM amplifies self-bias in self-refinement. In Lun-We Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computationall Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11 11-16, 2024, pp. 15474-15492. Association for Computational Linguistics, 2024. doi: 10.18853/V1/222 ACL-LONG.826.
Ling Yang, Zhaochen Yu, Tianjun Zhang, Minkai Xu, Joseph Gonzalez, Bin Cui, and Shuicheng Yan. Supercorrect: Supervising and correcting language models with error-driven insights. CoRR, abs/2410.09008, 2024. doi: 10.48550/ARXI..441.00000 48550/arXiv 2410..1100000
URL htttps://doii http::///oooooggg//// oorg/10 100
Weiran Yao, Shelby Heinecke, Juan Carlos Niebles. Zhiwei Liu, Yihao Feng, Le Xue, Rithesh R. N., Zeyuan Chen, Jianguo Zhang, Devansh Arpit, Ran Xu, Phil Mui, Huan Wang, Caiming Xiong, and Silvio Savarese. Retroformer: Retrospective large language agents with policy gradient
15

---

Published as a conference paper at ICLR 2026
optimization. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Seonghyeon Ye, Yongrae Jo, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, and Minjoon Seo. Selfee: Iterative self-revising IIm empowered by self-feedback generation. Blog post, May 2023.
Siyu Yuan, Zehui Chen, Zhiheng Xi, Junjie Ye, Zhengyin Du, and Jiecao Chen. Agent-r: Training language model agents to reflect via iterative self self--raining. CoRR, abs/2501.11425, 2025a. doi: 10. 48550/ARXIV.2501.1 1425.. httppt::://ooo wttp:/o...........0............. http://1000555///a/1a1// .tl.vv225551 2501 11425
Siyu Yuan, Zehui Chen, Zhiheng Xi, Junjic Ye, Zhengyin Du, and Jiccao Chen. Agent-r: Training language model agents reflect via iterative self-training. arXiv preprint arXiv:2501.11425, 2025b.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning. In Sanmi Koyejo, S. Mohamed, Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang. Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang. Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang. Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, and Xuanjing Huang. Secrets RLHF large language models part I: PPO. CoRR, abs/2307.04964 2023. doi: 10.48550/ARXIV230000 .04944
Xin Zheng, Jie Lou, Boxi Cao, Xueru Wen, Yuqiu Ji, Hongyu Lin, Yaojie Lu, Xianpei Han, Debing Zhang, and Le Sun. Critic-cot: Boosting the reasoning abilities of large language model via Shain-of-thougee critic, 2024.
16

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
LLMs are utilized in this manuscript for partial grammatical checks and language polishing, The authors are fully responsible for the final content.
B PERFORMANCE ON VARYING ACTOR MODELS WITH DIFFERENT CAPABILITY LEVELS
To further investigate the Critique-RL in varying base models, we conduct two types of ex- periments. In the first setting, we use a strong reasoning model -e-Seek--R--issill-- 7B (DeepSeek-Al, 2025) as our actor model while using Qwen2.5-7B as our critie model. This evaluation setting investigates the generalization of Critique-RI. to reasoning models. The results in Table S reveal that, be- sides non-rcasoning models (Qwen2.5- 3B, Qwen2.5-7B) with structured CoT,, method is also effective for reasoning models with complex CoT structures on both in-domain and out-of-domain tasks, particularly in terms of the Acc Dis achieved by the critique models. While -e-Seeck-R--istilll Hwllll--eenBBBB 7B already performs strongly on MATH-500, cri- tique models can still offer marginal gains reasoning accuracy. More im- pressively, on the TheoremQA dataset which spans diverse domains including Math, EECS, Physies and Finance, cri- tique models substantially boost perfor- mance, highlighting the strong general- ization ability of our approach. Notably, Critique-RL outperforms SFT, Retroformer, and CTRL by 26.75, 28.75, 29.88 points in Acc @ Dis, respectively, on the TheoremQA dataset-doublin the performance of these baselines.
Table 5: Performance on -i-iss-Ril--iti---i-e--e actor.
In-Domain: MATH-500
OOD: TheoremQA
Methed
Acc
A
Acc@
Acc
A
Ace Dis
No Critic
SFT
85.60
1.00
83.40
29.75
8.13
24.38
Retroformer
85.80 50.80
1.20
84.80
29.38
7.75
22.38
CTRL
85.80
1.20
84.80
Critique RI.
86.60
2.00
93.00
30.38
8.75
51.13
Table 6: Performance on Qwenn.55-7BB-Instteen as actor,
In-Domain: MATH-500
OOD: TheormQAA
Method
Acc
A
Acc& Dis
Acc
A
Accé Dis
No Critie
79.10
21.35
SFT
79.20
0.10
80.20
21.63
0.25
23.00
Retroformer
79.20
0.10
80.60
21.75
0.38
21.38
CTRL
79:40
0.30
79:40
21.50
0.13
21.13
CritiqueRL RL
80.30
1.20
89.20
23.50
2.10
46.63
In the second setting, we use 1et-m2.55-72BInstttre the actor model and Qwen2.5-7B as the critique model to investigate weak-to-strone generalization.
The results in Table 6 show that Critique-RL improves actor performance even in large-scale set- tings, though with less pronounced gains compared to smaller-actor settings. Nonetheless, it still outperforms baselines on both in-domain and out-of-domain tasks. Notably, our method achieves significantly higher discrimination, confirming the effectiveness of our discrimination-base reward shaping.
C PERFORMANCE ON VARYING MODEL SERIES
Table 7: Performance on Llama3.2-2 with GSM8K.
GSM8K
Method
Acc
A
Acc@ Dis
No Critic
49.28
SFT
50.80
1.52
68.11
Retroformer
52.08
2.81
63.85
CTRL 52.24
2.96
66.01
Critique-RL
52.99
3.72
75.04
To evaluate the effectiveness and generalization capability of Critique-RL. we conduct experiments using the Llama3.2-3B (Dubey et al.) 2024) model on the GSM8K dataset. As shown in Table 7 Critique-RL proves effective not only Qwen2.5 models but also Llama3.2 models, particularly
17

---

Published as a conference paper at ICLR 2026
in enhancing the discriminability of the critique models. These results highlight the adaptability and robust performance of Critique-Ra RL across different model architectures.
D PERFORMANCE ON MORE CHALLENGING BENCHMARKS
To further validate the effectiveness of Critique-RA RL, we conduct experiments using Qwen2.5-785 Instruct and evaluated on the AIME2024, AIME2025 (AIME) 2025), and GPQA-Diamond (Rein et al. 2024) benchmarks. We used General-Reasonees training of 30k examples based the WebInstruct- Verified (Ma et al. (025) dataset for RL training. The experimental results are in Table 8 The experimental results demonstrate that our method significantly improves the critique model's 's discriminability, with particularly notable improvements on the challenging reasoning datasets AIME2024 and AIME2025. Without fine-grained hyperparameter adjustments, our method outperforms the baseline across all three benchmarks, proving the effectiveness of Critique-RR RL in complex reasoning scenarios.
(Ma et al. 2025) as the actor and constructed
Table 8: Performance on challenging benchmarks using Qwen2.5-7B-Inss In-nttuctt as critic.
GPQA-Diamond
A1ME2024
A1ME2025
Methods
Acc
A
Acc@ Dis
Acc
Delta
Acc@ Dis
Acc
Acc@ Dis
No Critic
34.30
11.98
6.67
SFT 35.86
1.56
41.20
12.30
0.32
22.71
7.50
0.83
6.67
Critique-RL
37.37
3.07
51.52
13.75
1.77
53.44
8.50
1.83
30.10
E COMPARISON WITH OTHER IMPORTANT REFINEMENT METHODS
To further validate the advantages of Critique-RL RL. over other refinement methods, we conduct evaluations of other refinement methods including Self-Refine (Madaan et al.) (023), SuperCorrect (Yang et al. 2024) and Critic-Cot (Zheng et al. 2024) with Qwen2.5-3B on GSMSK. For a fairer comparison, we train the models in Self-Refine and Critic-CoT using the same dataset(sampled from Qwen2.5 3--3-IInntruc) as Critique-RR RL. In terms of SuperCorrect, we choose Deepseek-E R1 (DeepSeek- AIl 2025) the teacher model to create both the Hierarchical Thought Templates and positive critique datasets. The results are presented in Table 9 Critique-RL significantly outperforms all other methods in both Acc and Acc @ Dis, surpassing Critic-CoT and SuperCorrect by 5.31 and 3.11 points in terms of Acc, respectively. Moreover, Critique-RL. outperforms Self-Refine across refinement iterations, demonstrating discriminability. likely because it simply used teacher model data as positive examples and student model data as negative ones for DPO training. Given the GSM8K dataset's 's simplicity. the student model's 's output is not consistently inferior to teacher model's, ss leading to potential impairment to the model's s discriminability.
greater effectiveness. Notably, SuperCorrect exhibited poor
These refinement methods are implemented using SFT (Self-Refine), self-improv (Critic-CoT)) or intricate SFT+DPO (SuperCorrect) approaches, wheras Critique- employs an online RL methodol- ogy, which accounts for its observed performance advantages.
F MORE TEST-TIME SCALING RESULTS
The results of inference compute scaling GSM8K are illustrated in Figure Similar to the findings on MATH, Critique-RL is more compute-efficient and significantly increases the performance ceiling. validating the potential of our approach. In addition. we evaluate the refine compute scaling of SFT and Critique-RL RL across MATH, GSM8K, and AQUA, as illustrated in Figure 7 Critique-RL consistently achieves approximately twice the sampling efficiency of SFT. Notably, with the 7B model on GSM8K, Critique-RL' RRLss Pass even surpasses the SFT's Pass @ @64, demonstrating the effectiveness of approach.
18

---

Published as a conference paper at ICLR 2026
Table Comparison with other refinement methods with Qwen2.5-3B on GSM8K.
GSM8K
Method
Acc
Acc@ Dis
iteration=1
71.42
75.84
Self-Refine
iteration=2
72.71
76.52
Critic-CoT
70.58
74.70
SuperCorrect
72.78
62.17
Critique-RL (Ours)
75.89
87.44
Scaling Inference Compute: GSM8K
90
Majority Vote@k
80
75
w/ Critique-RI RL
70
w/o critic (@k)
wlo critic (@2k)
65
w/o critic (@3kk
2
2
23
2
2
26
2
Number of samples
Figure 6: Inference compute scaling for Critique-RL, with @2k and @3k indicating sampling amounts that are 2 times and 3 times the x-axis value, respectively. Critique-RL improves the performance ceiling and more compute-effiiiet.
G PERFORMANCE ON SUMMARIZATION TASK
For open-ended tasks where rule-based verifiers cannot be directly applied, reward signals can be provided through additional reward models or Al feedback (c.g., using GPT-4o (OpenAl 2023) for judgement).
conduct experiments of Critique-RL RL with Dowen..5-7B-Insttee (Team) 2024) on summarization task using CNNiDailyMaiil (Hermann et al., 2015) dataset. Specifically, given article x, the actor model generates an original summary y. The reward model (Skywork-Reww Reward- V2-Llama-3..1 1-BB (Liu et al. 2025 then evaluates the summary, with its output linearly scaled 1-10 range, i.e., Toracle ( y) Subsequently, the critique model produces critique c, which includes comments about the summary across key criteria, quality score from 1-10, and improvement suggestions. The actor model then generates a revised summary y' accordingly, which is also scored by the reward model to yield refinement score Trefine Toracee y'). Based on this, we define the discrimination reward function of the critique model as:
Fdis (x. c) max [f y. c) Tomale u)1
8
where f(x y, c)is the quality score of the original summary from critique model. 8 is the permissible maximum error range.
In stage we optimize the discriminability of the critique model using ras (ar y. c); In stage II, we optimize the helpfulness while maintaining discriminability using the following reward function:
staggel
Trefine + 3. v.
In our experiments, we select 5000 training and 1000 test queries from CNN/DailyMail 3.0.0's official splits. The results are presented in the Table
19

---

Published as a conference paper at ICLR 2026
Scaling Refine Compate:
Scaling Compute: GSMRK
Scaling Refine Compute: AQUA
75
75-
70-
65
65-
60-
80-
55-
55-
50
45
50-
2
2"
2
2
2
2
2
2
2
2
2
2.
2
2
2"
2
Number samples
of d'samples
Number sumples
Sit (B1
CL RLSB)
Si artec(7))
Chttque-RLL RL(78)
Figure 7: Refine compute scaling for Critique-RU Milgquee-Laaa and SFT critic with Qwen2.5-3B and Qwen2.5-7B.
The results reveal that Critique-RI. can effectively optimize discriminabil- ity, yielding improvement in summary quality. We use MSE and MAE to measure the enror between the qual- ity scores produced by the critique model and those from the reward model. Specifically, Critique-RL outperforms baseline by 0.87 points in Score, 7.87 points in MSE@ Dis and 1.79 points MAE@ Dis. These improvements demonstrate the strong generalization ability of our approach to open-ended tasks, contributing to scalable oversight.
Table 10: Performance on summarization task using Dwen2.5-7B-Instreee
We report the original Score by reward model. The MSE@ Dis stands for mean square er- ror, and MAE@ Dis stands for mean absolute error, where smaller values indicate stronger discrimination abilities.
Method
CNNMD
Score?
Delta?
MSE
MAE
Critic
19.89
7B-Instruct
19.91
0.25
9.46
27
Critique RL (Ours)
20.81
1.12
1.59
0.98
H VALIDATING THE EFFECTIVENESS OF CRITIQUE MODEL
Introducing separate critique model leads to increased manual effort and additional complexity. To validate the usage of the critique model, we compare Critique-RI. with actor-only RI. method to show that training critique model provides significant benefits over directly optimizing the actor. In particular, for actor-only method, we conduct experiments on directly RL the actor and SCoRe (Kumar al. 2024); for actor-critic paradigm, we use SFT-based critique model well as Crituqe-RL RL. For a fairer comparison, we train the actor model using the same reasoning traces as Critique-RL RL in direct RL and using the same reasoning, critique and refinement dataset as Critique-RI RL in SCoRe. All experiments are conducted with Qwen2.5-7B on the Math dataset.
Table 11: Comparison with actor-only RL method.
MATH
Category
Method
Acc
Ace@ Dis
Directly RL
49.78
Actor-only
SCoRe
56.52
72.51
SFT
51.84
67.59
Actor-Critique
Critique-RL
58.40
85.20
The results in Table show that Critique-RI significantly outperforms Directly RL by 8.62 points in terms of Acc. Also Critique-RL outperforms SCoRe by 12.69 points in terms of Ace@ Dis, and 1.88 points in terms of Acc. Note that during the training process of Critique-RL, the actor model remained fixed and thus inherently weaker reasoning and refinement than the trained SCoRe actor model. Importantly, the trained critique model can be flexibly applied to other stronger actor models (weak-to-strong. to-strong) and reasoning models to further improve their performance(sce Section 6). This modularity and transferability are advantages that SCoRe lacks.
20

---

Published as a conference paper at ICLR 2026
Moreover, conduct the test-time scaling experiment. The majority vote (MV@K) results are shown in Table 12 The results show that even the actor model has been well rell-trained, generating parallel responses still underperforms Critique-RLe RL's response-critiquee critique-refinemen process. Notably, Critique-RL's MV @ even surpasses Directly RL's MV@ 12. This highlights the compute-efficiency of Critique-RI.
Table 12: Performance comparison between Directly RL and Critique-RL under MV @ K.
Directly RL
Critique-RL MV @ K
K
MV@K
MV @ 2K
MV@3K
49.78
50.05
52.39
58 68.40
2
50.05
53.49
55.04
59.10
53.49
55.08
56.75
65.91
I SENSITIVITY ANALYSIS
For solidness, we provide details about different values for 3, $1, 32 and training steps per stage.
Experiments on different values for 8, f1, and 32. We exemplify our selection of the parameters 8, $1, and B2 by presenting the performance of the Qwen2.5-3B model the GSM8K dataset as an example. The results Table reveal that these parameters are not sensitive, so we ultimately choose 3 0.01, 31 0.9, and 32 0.95 for our experiments.
Experiments on different training steps per stage. We show the performance of the two stages of Critique-RL at different training steps with Qwen2.5-3B on MATH dataset. The results in Table indicate that within 500 steps of Stage I, the model's discriminability was substantially enhanced, with Acc@ Dis rising from 66.51 78.68. During Stage II, the model maintained this discriminability while further improving helpfulness, with Acc increasing from 45.90 to 48.60.
While further refinement of parameters could potentially yield additional performance gains, the current experimental outcomes are already statistically sound and adequately substantiate our core conclusions.
J QUALITATIVE ANALYSIS
We perform qualitative investigation into how Critique-RI. works and provide several examples in Appendix J In Figure 8 facing the originally incorrect response, the critique model after SFT is unable to detect errors, leading the actor's refinement response to retain the same errors. However, the model trained after Critique-Re RL identifies the errors in the original response and provides detailed, constructive suggestions for modification, leading to the correct refinement response. In Figure 9 model trained after Critique-RL Stage I is able to detect errors, demonstrating its discriminability. However, the model provides the actor with low-quality suggestion, causing the actor's refinement response to be incorrect. In contrast, for the same erroneous original response, model trained after Critique-RL Stage II not only detects the error but also offers constructive suggestion, ultimately leading to the correct refinement response, demonstrating the advantage of two-stage RL process.
To directly assess the quality of critiques generated by Critique-RL, we randomly collect 600 critiques that successfully helped refine incorrect answer into correct ones. We leverage GPT-4o with ground- truth answers and solutions as references to evaluate quality more accurately. The results show that 96.2% of these critiques made correct discriminative judgments, and 93.3% were rated as high-quality, demonstrating that Critique-R RI, produces reliable and helpful critiques.
21

---

Published as a conference paper at ICLR 2026
Table 13: Results of different values for B, 31. and 32 with Qwen2.5-3B on GSM8K.
Parameter
Value
Acc
Delta
Ace@ Dis
0.008
74.60
8.57
86.24
B
0.01
75.89
9.86
87.44
0.012
74.22
8.19
87.10
0.88
74.60
8.57
86.18
0.9
75.89
9.86
87.44
0.92
74.68
8.65
86.09
0.93
74.68
8.65
85.99
82
0.95
75.89
9.86
87.44
0.97
74.37
8.34
85.74
Table 14: Results of different training steps per stage with Qwen2.5-3B MATH.
Stage Critique-RI
Critique-R.
R.aeeRRR.SSaaa Stage II
Step
Acc
Acc@ Dis
Acc
Ace@ Dis
0
44.24
66.51
45.90
78.68
100
44.22
68.26
45.88
80.56
200
44.60
71.53
46.82
81.77
300
44.89
75.72
47.02
82.47
400
45.18
78.20
47.90
83.06
500
45.90
78.68
48.60
82.80
22

---

Published as a conference paper at ICLR 2026
Problemn: Each
brusehotd comsenaas riee Suppostng members houstoold rioe lanch and dinner, weeks will kig
ofe bet?
Ground Truth Answer:
Actor Model (original response);
Each person eas 0.2 lignneal x coeasddyy - -00,22200 bot kig ofiice
Lets d-dow sop skyy haaachold
per den.
So, the 5 roenter conteme hepoo 5 people oog (1dedood
permeal pur
lg rise per day.
the bozscholdd comume 0.4 ka of rize per day 5 wodk in k of of wock.
kgdday daystweek
oxg-7- 1dbol flg
A.A. lg bea ofe will las kg la week 21 wedlo.
The reswer 3.
Uhas, lgg tas will las kg koplwokk 0o42214--0oo warks.
The arewer
Virifier:
Critique Model after SFT
Critique Model after Critique-RL Training
Critique Model (critique): 114 kg rio per
Crilique Model (crilique);
bouchold
0.2 of rice
Stepaentoner looochol consurtes 0.2 kg of
per day kg
Eaphiaaiion: This sag simaalyy caloziato the daily ries crnampiion by mashpphynng the amoont cossomed per meal (0. kg) the sumter of moets per (2).
Conremees the stegs Correct
Carnettens step: Wreng
Explanation: This
wazea becazee inomectty cakodaazs the daily cizz
crnampptinn Siree 5 murnhere of hosadhid d mus every kunch and dimmar onch passe cossezes 0.2 lg thee per nedl Be daily cossumption is:
memberrs x mich//oy x fegimol kaplaa
weak 2 kg of louschold werk.
lig of rioe day
The truadsiil
day per wark
Eaplunation: This sap camastly calodiato the weakly rice conmumptiin by mactppying the dilly conecrytion (1-4 kgg by the number of days a work (5).
Comeemes of the stegi Cornect
Cornettew of the steg: Weseg Explanation: This rapp is wrin becuues it treemetts caleritass the weakly rie consinition. Gives the conect dely consunptton is tigling the weekly consunption
week.
2agdday daass/ookk ka./week
oiee las 42 la wark wedus Lurrecmos the stops Currect
42
lad
Explonation: This comuutly caloutasee the nanther of weeks the 43 bg bog ofe will las by droding the load anounn one by the workly comsumption le por worr)).
Curnetness stop: Wreng
Explanation: Tii ska enen, benane il insemedly cale altto lhe Carniien the rise will aus Given the cormes wuecly consarption is 14 sigheekk, the dimmtonn the 42 bag will lest
82 = 34 loghoooii in Siwooks
proside
Explonation:
Discriiiaaiion Veriffer:
Explonation: The fioi anwoer of wakke invemnnt The cormar anower Ic Tuadkr
Cernectocn final
Discrinniation Verifier:
Actor Model (refinement responsej:
Actor Model (refinement response):
Caleulite the daily siee massumption: The boodhold coesanes 0g of tha per med. Sinae borc en maah par day. the daily conamppies is: 0.2 ag/neal x meatc/day = 0-4 lig/try
Lat' break down by
TIn deremine how wekk beg ofrccee will heeahoid d, need comsddr the fillwiigg
1. thity Rise Conummptin Calculadoon Thrn so marthrr is the sazsahetd Frach memeber comouree 0.3 la of per neal Eich sember eit rice at moek per dar (luxh and diszer). Threcies
Caleubste weekly ries consumption: bocachcld comamm 0.4 lig par
lidday dayssweekk alg/week
conoumption
Sineteess meals/day
dully nas coenzmation
lighneal 2lgg/lay
1. Calembse
42 ofee will laa. A
Weskdy Thenbio wecity rice cocsurption i6
cxcuuppiin kg
iie will lest
The seower 33.
82 2 leock 21
Duration the Rice will Last: The hoswheld he lop beg ofrice The woakly consunaptin is kg Theecfore the namber of webls the iie sill kai
2 ligilays? waokk 36 la/wwodi
lgghoook weakss Correctness Verifierr
Correctness Verlflerr
The answer 3
Figure& 8: Example of qualitative analysis. The actor's original response is incorrect. The model after SFT is unable to detect errors in the response, leading the actor's refinement response to retain the same errors. However, the model trained after Critique-RL identifies the errors in the original response and provides detailed, constructive suggestions for modification, leading to the correct refinement response.
23

---

Published as a conference paper at ICLR 2026
Prublem: Denri pound of ofl th
doean criounta reako. He nodé cniiuuts. The groerry curcelly prerocizn luy poand
poandh better?
Grand Truth Answer:
pound of busie makes I dost cnisuano and need make dooes
be needs 166 oct toppod peonds of honer
Actor Model (original response):
The buta ia 54,00 pound caneall enoale for buy une as one fulf off evry cat of perzad of turner will coet 6*1
Lat's troak down any by ary: pousss lecase --3-
$o0411.5 (0000-6000
The sile peice 56.0 2 pouada of buter and he needs buy usiaa bate on pes S-ogrt- 18 18.00 wort of hasee
weds buy -okd--o0 1ood unite worh hestirr ofherer bentrr
of2
The snower ic
be will
The reswe 18.
Currectass Verifier
Critique Model after Critique-RL Stage I Training
Critique Model after Critique-RL Stage II Training
Critique Model (critique):
Critique Model (critique):
Step smitene:: noad bey pounde butter because 6/2
Correntes of the stepe Wneng
Correturs
Wreeg
Explasatise: Detzia scod to make 5 doee cocavantl, and be cass pound of haker perdonor. Therskms, reads pf hoates The caloration 3 incerret reeds buy pounds
make 6 dooe uniiuuso, and saxe pound of butker deeas Therefor, he neds poends of o'hhree The pronecioon does not change the fact that le needa prznds. The calouetion 6 3 inclevant to ise problan.
Exphiaatio:: Dentis
Siep
would $12 beassse 3 4--
vatmee pranda butler step:
and 512 boxame
Explonation: Aithough the oile zlatio the coe locomeet poonde at Si per poard couls 5125, the premiis is wony, Denzia ccods 6 pounds of buttos rex oirma caloration ahould porrdcan por pound, which equal 524.
Corremaos
Explonation Alboogg pronetionn is
pousd oohere,
half Dentis
roah pazsk butes The purrotion aprice
poundl hup,
pounds le pave St - 53-53 St. Stroo needs 6 pounds he can tiko acharage ofth promecion 5 lanca poundal posnds per poonodions. Therefore Zoe soaal cosi is 56
Correstaos step: Wevsz
Step Ho
becose 12/2
12/2-a.
Explonation: The coleabe 12/3 6i correct hosed the perestion (hey soe poand get une balf olly, but is does not spple to the curret ansor of huter rendel Dernic nort pounah, sei.. The cormet caksiaaiin duoild grke goonds
Coneetuoss
Wreeg
Explination: The calcalatia "12/ 2- 6f inceerees Hasod previoza explanation. The ssod cred ic X 56- S1S. Thanstinn he will pay $18 Inneadd of
basel
Correctues the final aneweer Werng Exphoatiion:
Correctuess fixsl
Werng
assvece of incorcet Gives gramotionn Dessis will
Explanation: The find msowc of insematt Besed un the corcel Rioillatios, Dredts 38dd 5 pounds of botte at pic poond seich SSA
Diseriaiination Verifier:
Actor (refinement response):
Actor Model (refinement responsej:
Diserinination Verifier:
lecak down by deps
Lats lmak & down dep says
To drsmmiiee how much it will cos Danrle to perchss poonde of hhrreer we real to crnsde the Gelloiigg
cossle Sillwwing
Butter Requirement: Denzis soab make 6 duze eniasanb, and pound hetur per donce. Thaufinns raaded of heraer
parchas pounds
1. mutter Heggttement: Dernic moek make donun exisarts, and une dezas.
needs poands oftutter
Promotion Detaib:: The promodzze is "buy one pound of buttes art oce hulf off!" This mare every 2 pourch haster parshwed, Denats pays for poord price and gate te secead parad
boner
prio.
pric.
& Cast Calcelation:
Fall Price fors Prande since the prometion spplic to euch of
For every
benoe, Doneis St for the fine
for sececi
pounds we aced caloetaee the east booe on be full price for poasds
pounal bealing 56 Sn pounts
Cat a Prund:: The cre poundica
Thancton
po
per aound 336)
Since Dereaa neade à pounds can take adhrrage tha promocion 25 tines 16 soznza
Thession:
pounds per promodizn)))
S18
Coerectness Veriffirr
Carrectuess Verlfierr
Figure 9: Example 2 of qualitative analysis. The actor's original response is incorrect. The model trained after Critique-RL Stage is able to detect this error, demonstrating its discriminability. However, the model provides the actor with low-quality suggestion, causing the actor's 's refinement response to be incorrect. In contrast, for the same erroneous original response, model trained after Critique-RL Stage II not only detects the error but also offers constructive suggestion, ultimately leading to the correct refinement response, demonstrating the advantage of two-stage RL process.
24


Reviews:
[{"review_id": "7p8Pygk9a5", "reviewer": "Reviewer_4UWL", "summary": "This paper proposes an online RL approach called Critique-RL for developing critiquing language models without stronger supervision. This approach contains a two-player paradigm, where the actor generates a response, the critic provides feedback, and the actor refines the response accordingly. The authors devise a two-stage optimization strategy, where stage I reinforces the discriminability of the critic with direct rule-based reward signals and stage II introduces indirect rewards based on actor refinement to improve the critic’s helpfulness. Experimental results show the effectiveness of Critique-RL.", "questions": "I have included my questions in the weaknesses part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed two-stage RL method is sound and well-motivated, which deals with the core problem of critique generation.\n2. Extensive experiments show the effectiveness of the proposed method.\n3. This paper is overall well-written and easy to follow.", "weaknesses": "1. The design of indirect rewards based on actor refinement is similar to [1], which is not discussed in the current paper. The authors should further clarify the difference between this work and [1] to highlight their novelty.\n\n2. The quality of generated critiques should be individually measured via automatic metrics or human evaluation.\n\n\n[1] Training Language Model to Critique for Better Refinement. ACL 2025 Findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gGwYweGnGZ", "reviewer": "Reviewer_WXJP", "summary": "This paper proposes Critique-RL, a two-stage reinforcement learning (RL) approach to train critiquing language models without requiring stronger supervision. The authors first show that baseline RL methods, which use only indirect rewards from an actor's refinement, fail. This is because they improve the critic's helpfulness (constructive feedback) but not its discriminability (judging correctness), leading to poor performance. Critique-RL solves this by:\n- Stage I: Explicitly optimizing discriminability using direct, rule-based reward signals.\n- Stage II: Optimizing helpfulness using indirect rewards (actor refinement) while using regularization to maintain the discriminability from Stage I.\n\nThis two-stage strategy delivers performance improvements on both in-domain and out-of-domain tasks, e.g., +9.02% in-domain and +5.70% OOD for Qwen2.5-7B.", "questions": "The paper is motivated as an approach for training critique models \"without stronger supervision\". However, the method's crucial first stage relies entirely on a direct reward $r_{dis}$ from an \"oracle reward function\" $r_{oracle}(x,y)$. This oracle, which knows the ground-truth correctness, seems to be a form of strong supervision.\n\nCould you please clarify this apparent contradiction?\n\n1. How do you formally define the \"stronger supervision\" (which you avoid) versus the \"oracle verifier\" (which you use)?\n2. The paper's contribution seems to be a novel way to distill the knowledge of a \"weak\" binary oracle (answer-checker) into a \"strong\" generative critic (feedback-generator). Would you agree with this framing?\n3. How does this framework scale to complex domains (e.g., creative writing, complex coding) where no such simple oracle or high-quality reward model exists?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper's core originality is its clear diagnosis of a key failure mode in training critics: baseline RL methods create a conflict between \"discriminability\" (judging correctness) and \"helpfulness\" (providing feedback), optimizing the latter at the expense of the former.\n- The paper's quality is high, with a rigorous methodology. The training dynamics in Figure 3 clearly show the baseline's failure , while decisive ablation studies in Table 3 prove that both stages of Critique-RL and its specific regularization are essential for success.\n- The work significantly contributes to scalable oversight by providing an effective method to train critics without stronger supervisors. Its value is shown through broad applicability, including \"weak-to-strong\" generalization (a 7B critic improving a 72B actor) and effectiveness on OOD and open-ended tasks .", "weaknesses": "- The paper's primary motivation is to train critics \"without stronger supervision\"1. However, the entire method, especially the critical Stage I, is heavily reliant on an \"oracle reward function\" $r_{oracle}(x,y)$ to compute the direct discrimination reward $r_{dis}$. For the main experiments on math tasks, this oracle is a rule-based verifier that knows the correct answer. This oracle is a form of strong, external supervision.\n- The framework's success, particularly in Stage II, hinges on a critical assumption: the fixed actor model $\\pi_{\\theta}$ is already a good \"refiner\". The authors state the actor is pre-trained to be \"capable of... faithfully refining them according to critiques\". This assumes away a large part of the problem. The helpfulness reward $r_{refine}$ is a convolved signal of both the critique's quality and the actor's ability to understand it. If the actor is a poor refiner, $r_{refine}$ becomes a noisy or meaningless signal, and Stage II would fail to optimize for helpfulness.\n- The paper correctly highlights its inference-time compute-efficiency benefits (e.g., in Figure 1 and Figure 6). However, it completely omits the training-time cost of this complex, two-stage RL pipeline. Stage II, for example, requires at least three model forward passes per training sample (one for the critic $c=\\pi_{\\phi}(x,y)$, one for the actor's refinement $y^{\\prime}=\\pi_{\\theta}(x,y,c)$, and one for the oracle/RM $r_{refine}=r_{oracle}(x,y^{\\prime})$). This is significantly more expensive than the SFT or baseline RL methods it's compared against.\n- The paper's core insight is that final-outcome rewards are insufficient, and a more direct signal is needed. The proposed solution, $r_{dis}$, is a direct reward for judging the outcome of the original response. This overlooks a more direct comparison to Process-based Reward Models (PRMs), which provide supervision at each step of the reasoning. The qualitative examples (Figs. 8, 9) show the critic is evaluating step-by-step, but it is only trained on the final answer's correctness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3FwnnFcE3N", "reviewer": "Reviewer_ao7e", "summary": "The paper addresses the challenge of insufficient scalable supervision for LLMs in complex reasoning tasks. It proposes Critique-RL, a two-stage reinforcement learning framework without strong supervision. In the first stage, the critique model’s discriminative ability is optimized using direct rule-based rewards. In the second stage, the framework integrates indirect rewards from agent-corrected response accuracy with regularization to enhance the usefulness of feedback while preserving discriminative capacity. Critique-RL achieves  performance improvements over baselines on several mathematical reasoning and question-answering tasks.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed two-stage RL method is effective to provide constructive critique feedback for better refinement and precise filter for effective time-time scaling.\n- The experiments contain several benchmarks across different tasks.\n- This paper is well-written and easy to follow.", "weaknesses": "My main concern lies in the experimental design, as I am not fully convinced that the current experiments sufficiently demonstrate the proposed method’s advantage on complex reasoning tasks.\n\n- Since the authors explicitly state in the Abstract and Introduction that their method targets complex reasoning tasks, more challenging benchmarks such as AIME and GPQA should have been included in the evaluation.\n\n- Although the main text reports significant improvements on Qwen-3B and Qwen-7B, the appendix reveals that the performance gains on stronger models, such as DeepSeek-R1-Distill-Qwen-7B and Qwen2.5-72B-Instruct, are quite limited. These results should be reported and discussed in the main paper.\n\n- The proposed method requires access to ground-truth answers to compute rewards. Under this setting, it remains unclear what advantages it offers over directly training the model’s reasoning ability using RLVR methods (e.g., GRPO, DAPO). Additional experiments are needed to clarify this distinction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qqfXckyIxe", "reviewer": "Reviewer_FEHf", "summary": "The paper proposes Critique-RL, a two-stage reinforcement learning (RL) framework for training critiquing language models. The critic  models that assess the correctness of an actor’s response and provide natural language feedback to guide refinement. The core motivation is to avoid reliance on stronger supervisors for critique annotation, which is costly and hard to scale. The key insight is that optimizing solely via indirect rewards, whether the actor’s refined output is correct, leads to poor discriminability, the critic’s ability to accurately judge whether an initial response is correct. To address this, Critique-RL introduces a two-stage training process. Experiments on mathematical reasoning benchmarksshow consistent improvements over baselines.", "questions": "No question", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem setup, motivation, and methodology are clearly articulated with intuitive figures (e.g., Figure 2, 3) and precise definitions of evaluation metrics\n2. Scalable oversight remains a critical bottleneck in LLM alignment and self-improvement. By explicitly disentangling and jointly optimizing two key critique capabilities, the work offers a practical and empirically validated pathway toward more reliable critique models.\n3. The identification of problem due to conflicting objectives under indirect rewards is compelling.", "weaknesses": "1. The paper claims to avoid “stronger supervisors” for critique data, yet relies heavily on ground-truth answers (i.e., golden labels) to construct both direct (Stage I) and indirect (Stage II) reward signals. These labels are typically human-annotated or derived from curated datasets (e.g., MATH, GSM8K). Thus, the method still depends on human-provided supervision, albeit not in the form of natural language critiques. This undermines the central motivation that existing approaches “rely on stronger supervisors for annotating critique data”—since here, human-labeled answers serve as an equally strong (if not stronger) form of supervision.\n2. Lack of analysis of some highly related works in this paper [1], which also proposes to use the refinment as the additional feedback.\n\n> [1] Training Language Models to Critique With Multi-agent Feedback", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)