RefCritic: Training Long Chain-of-Thought Critic Models with Refinement
  Feedback
Authors: name: Qiaoyu Tang; name: Hao Xiang; name: Le Yu; name: Bowen Yu; name: Hongyu Lin; name: Yaojie Lu; name: Xianpei Han; name: Le Sun; name: Junyang Lin
Published: 2025-07-20 16:19:51

Abstract: With the rapid advancement of Large Language Models (LLMs), developing
effective critic modules for precise guidance has become crucial yet
challenging. In this paper, we initially demonstrate that supervised
fine-tuning for building critic modules (which is widely adopted in current
solutions) fails to genuinely enhance models' critique abilities, producing
superficial critiques with insufficient reflections and verifications. To
unlock the unprecedented critique capabilities, we propose RefCritic, a
long-chain-of-thought critic module based on reinforcement learning with dual
rule-based rewards: (1) instance-level correctness of solution judgments and
(2) refinement accuracies of the policy model based on critiques, aiming to
generate high-quality evaluations with actionable feedback that effectively
guides model refinement. We evaluate RefCritic on Qwen2.5-14B-Instruct and
DeepSeek-R1-Distill-Qwen-14B across five benchmarks. On critique and refinement
settings, RefCritic demonstrates consistent advantages across all benchmarks,
e.g., 6.8\% and 7.2\% gains on AIME25 for the respective base models. Notably,
under majority voting, policy models filtered by RefCritic show superior
scaling with increased voting numbers. Moreover, despite training on
solution-level supervision, RefCritic outperforms step-level supervised
approaches on ProcessBench, a benchmark to identify erroneous steps in
mathematical reasoning.