Critique-RL: Training Language Models for Critiquing through Two-Stage
  Reinforcement Learning
Authors: name: Zhiheng Xi, name: Jixuan Huang, name: Xin Guo, name: Boyang Hong, name: Dingwen Yang, name: Xiaoran Fan, name: Shuo Li, name: Zehui Chen, name: Junjie Ye, name: Siyu Yuan, name: Zhengyin Du, name: Xuesong Yao, name: Yufei Xu, name: Jiecao Chen, name: Rui Zheng, name: Tao Gui, name: Qi Zhang, name: Xuanjing Huang
Organization: Fudan NLP Lab

Training critiquing language models to assess and provide feedback on model
outputs is a promising way to improve LLMs for complex reasoning tasks.
However, existing approaches typically rely on stronger supervisors for
annotating critique data. To address this, we propose Critique-RL, an online RL
approach for developing critiquing language models without stronger
supervision. Our approach operates on a two-player paradigm: the actor
generates a response, the critic provides feedback, and the actor refines the
response accordingly. We first reveal that relying solely on indirect reward
signals from the actor's outputs for RL optimization often leads to
unsatisfactory critics: while their helpfulness (i.e., providing constructive
feedback) improves, the discriminability (i.e., determining whether a response
is high-quality or not) remains poor, resulting in marginal performance gains.
To overcome this, Critique-RL adopts a two-stage optimization strategy. In
stage I, it reinforces the discriminability of the critic with direct
rule-based reward signals; in stage II, it introduces indirect rewards based on
actor refinement to improve the critic's helpfulness, while maintaining its
discriminability via appropriate regularization. Extensive experiments across
various tasks and models show that Critique-RL delivers substantial performance
improvements. For example, it achieves a 9.02% gain on in-domain tasks and a
5.70% gain on out-of-domain tasks for Qwen2.5-7B, highlighting its potential.