Reinforcement Learning with Rubric Anchors
Authors: name: Zenan Huang; name: Yihong Zhuang; name: Guoshan Lu; name: Zeyu Qin; name: Haokai Xu; name: Tianyu Zhao; name: Ru Peng; name: Jiaqi Hu; name: Zhanming Shen; name: Xiaomeng Hu; name: Xijun Gu; name: Peiyi Tu; name: Jiaxin Liu; name: Wenyu Chen; name: Yuzhuo Fu; name: Zhiting Fan; name: Yanmei Gu; name: Yuanyuan Wang; name: Zhengkai Yang; name: Jianguo Li; name: Junbo Zhao
Published: 2025-08-18 10:06:08

Abstract: Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a
powerful paradigm for enhancing Large Language Models (LLMs), exemplified by
the success of OpenAI's o-series. In RLVR, rewards are derived from verifiable
signals-such as passing unit tests in code generation or matching correct
answers in mathematical reasoning. While effective, this requirement largely
confines RLVR to domains with automatically checkable outcomes. To overcome
this, we extend the RLVR paradigm to open-ended tasks by integrating
rubric-based rewards, where carefully designed rubrics serve as structured,
model-interpretable criteria for automatic scoring of subjective outputs. We
construct, to our knowledge, the largest rubric reward system to date, with
over 10,000 rubrics from humans, LLMs, or a hybrid human-LLM collaboration.
Implementing rubric-based RL is challenging; we tackle these issues with a
clear framework and present an open-sourced Qwen-30B-A3B model with notable
gains: 1) With only 5K+ samples, our system improves by +5.2% on open-ended
benchmarks (especially humanities), outperforming a 671B DeepSeek-V3 model by
+2.4%, while preserving general and reasoning abilities. 2) Our method provides
fine-grained stylistic control, using rubrics as anchors to mitigate the
"AI-like" tone and produce more human-like, expressive responses. We share key
lessons in rubric construction, data selection, and training, and discuss
limitations and future releases.