Title: Beyond Accuracy: Measuring Reward Variance as a Predictive Benchmark for RLHF
Primary area: datasets and benchmarks
Keywords: Reward Modal, LLM Evaluation, Benchmark, Reward Variance

Abstract: Reward models (RMs) provide the core signal in reinforcement learning from human feedback (RLHF). However, most evaluations focus on pairwise accuracy and overlook how the separability and concentration of reward distributions shape the optimization landscape and convergence rate. To address this limitation, we present the Reward Variance Benchmark (RVB), an evaluation suite that quantifies distributional properties of RM scores. RVB introduces three types of variance-oriented metrics that capture complementary aspects of an RM’s signal: score distribution concentration, global pairwise separation, and cross-prompt decision-style stability. We evaluate 23 widely used RMs with a toolkit that supports reproducible analysis. The metrics yield stable rankings and show a strong positive correlation with faster convergence in downstream RLHF training. The findings support a key insight: An effective RM must not only judge responses accurately, but also score them clearly to provide actionable gradients. RVB achieves this goal by adding a variance-centric dimension that aligns more closely with the RLHF optimization dynamics and offers empirical guidance for selecting and developing better RMs.

TL;DR: RVB shows that variance in RM scores—concentration, separation, stability—predicts faster RLHF convergence; clear, separable scores matter beyond accuracy.