Enabling Scalable Oversight via Self-Evolving Critic
Authors: name: Zhengyang Tang; name: Ziniu Li; name: Zhenyang Xiao; name: Tian Ding; name: Ruoyu Sun; name: Benyou Wang; name: Dayiheng Liu; name: Fei Huang; name: Tianyu Liu; name: Bowen Yu; name: Junyang Lin
Published: 2025-01-10 05:51:52

Abstract: Despite their remarkable performance, the development of Large Language
Models (LLMs) faces a critical challenge in scalable oversight: providing
effective feedback for tasks where human evaluation is difficult or where LLMs
outperform humans. While there is growing interest in using LLMs for critique,
current approaches still rely on human annotations or more powerful models,
leaving the issue of enhancing critique capabilities without external
supervision unresolved. We introduce SCRIT (Self-evolving CRITic), a framework
that enables genuine self-evolution of critique abilities. Technically, SCRIT
self-improves by training on synthetic data, generated by a contrastive-based
self-critic that uses reference solutions for step-by-step critique, and a
self-validation mechanism that ensures critique quality through correction
outcomes. Implemented with Qwen2.5-72B-Instruct, one of the most powerful LLMs,
SCRIT achieves up to a 10.3\% improvement on critique-correction and error
identification benchmarks. Our analysis reveals that SCRIT's performance scales
positively with data and model size, outperforms alternative approaches, and
benefits critically from its self-validation component.