Chasing the Tail: Effective Rubric-based Reward Modeling for Large
  Language Model Post-Training
Authors: name: Junkai Zhang, name: Zihao Wang, name: Lin Gui, name: Swarnashree Mysore Sathyendra, name: Jaehwan Jeong, name: Victor Veitch, name: Wei Wang, name: Yunzhong He, name: Bing Liu, name: Lifeng Jin
Organization: Scale AI

Reinforcement fine-tuning (RFT) often suffers from reward
over-optimization, where a policy model hacks the reward signals to achieve
high scores while producing low-quality outputs. Our theoretical analysis shows
that the key lies in reward misspecification at the high-reward tail: the
inability to reliably distinguish Excellent responses from merely Great ones.
This motivate us to focus on the high-reward region. However, such tail
examples are scarce under the base LLM. While off-policy exemplars (e.g. from
stronger models or rewrites) are easier to obtain, naively training on them
yields a misspecified reward for the policy we aim to align. To address this,
we study rubric-based rewards. By design, rubrics can leverage off-policy
examples while remaining insensitive to their artifacts. To elicit rubrics that
capture the high-reward tail, we highlight the importance of distinguishing
among great and diverse responses, and introduce a workflow to implement this
idea. We empirically demonstrate that rubric-based rewards substantially
mitigate reward over-optimization and deliver effective LLM post-training
improvements. Our code can be accessed at
https://github.com/Jun-Kai-Zhang/rubrics.git .