Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training
Authors: Junkai Zhang
Zihao Wang
Lin Gui
Swarnashree Mysore Sathyendra
Jaehwan Jeong
Victor Veitch
Wei Wang
Yunzhong He
Bing Liu
Lifeng Jin
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Post-training
Reinforcement Learning
Alignment
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement fine-tuning (RFT) often suffers from reward over-optimization, where a policy model hacks the reward signals to achieve high scores while producing low-quality outputs. Our theoretical analysis shows that the key lies in reward misspecification at the high-reward tail: the inability to reliably distinguish excellent responses from merely great ones. This motivate us to focus on the high-reward region. However, such tail examples are scarce under the base LLM. While off-policy exemplars (e.g. from stronger models or rewrites) are easier to obtain, naively training on them yields a misspecified reward for the policy we aim to align. To address this, we study rubric-based rewards. By design, rubrics can leverage off-policy examples while remaining insensitive to their artifacts. To elicit rubrics that capture the high-reward tail, we highlight the importance of distinguishing among great and diverse responses, and introduce a workflow to implement this idea. We empirically demonstrate that rubric-based rewards substantially mitigate reward over-optimization and deliver effective LLM post-training improvements.

Published as a conference paper at ICLR 2026
CHASING THE TAIL: EFFECTIVE RUBRIC-BASED RE- WARD MODELING FOR LARGE LANGUAGE MODEL PRAIINGGINING
Junkai Zhang" University of California, Los Angeles
Zihao Wang' Scale AI, Inc.
Lin Gui"
University of Chicago
Swarnashree Mysore Sathyendra
Jachwan Jeong Scale AI, Inc.
Victor Veitch University of Chicago
Scale AI, Inc.
Wei Wang University of California, Los Angeles
Yunzhong He Scale AI, Inc.
Bing Liu Scale AI, Inc.
Lifeng Jin Scale AI, Inc.
ABSTRACT
Reinforcement fine-tuning (RFT) often suffers from reward over-optiinizatinn where policy model hacks the reward signals to achieve high scores while pro- ducing low-quality outputs. Our theoretical analysis shows that the key lies in re- ward misspecification at the high-reward tail: the inability to reliably distinguish excellent responses from merely great ones. This motivate high-reward region. However, such tail examples are scarce under the base LLM. While off-policy exemplars (e.g. from stronger models rewrites) are easier obtain, naively training on them yields a misspecified reward for the policy we aim to align. To address this, we study rubric-based rewards. By design, rubrics can leverage off-policy examples while remaining insensitive to their artifacts. To elicit rubrics that capture the high-reward tail, we highlight the importance of distinguishing among great and diverse responses, and introduce workflow to implement this idea. We empirically demonstrate that rubric-based rewards substantially mitigate reward over-optimization and deliver effective LLM post- training improvements
focus on the
INTRODUCTION
In this paper, we are interested in how to produce reward models that are effective when used for L.M post-training. A reward model is function that takes prompt and response and produces a score quantifying how good that response for the prompt. post-training, we then align a lan- guage model to the reward by reinforcement learning type procedure. The fundamental challenge here is that, in many settings, it is nearly inevitable that the reward model will be an imperfect proxy for the behavior that we are actually trying to induce. In particular, this means that as we run post- training. it will increasingly be the case that the LLM is aligned to the idiosyncratic misspecification of the reward rather than the true signal that we are trying extract. this paper, we are interested in mitigating this effect.
Given that some misspecification is inevitable, what should we focus on when defining a reward model? The basic setup of post-training aims to induce the good behavior encoded by the reward while minimally shifting other aspects of the base LLM. Mathematically, this can be formalized
"Equal contribution.
Work done during internship Scale AL. "Our code can be accessed at http:///gthhh..oommmmmmka//aaann

---

Published as a conference paper at ICLR 2026
Off policy responses to capture the high-reward tail
rubric-based rewards to correct off-policy bias
Figure 1: Chasing the Tail with Rubric- Rewards
looking for post-training procedures that move along the Pareto frontier of KL divergence from the base model vs win rate (as judged by the reward) against the base model. We begin by theoreti- cally demonstrating that, for such Pareto-optimal procedures, the effect of reward misspecification is dominated by errors in the high-reward region. In other words, what really matters for post-training the ability accurately distinguish between the very good responses.
Then, we know that we want to focus our reward modeling on the high-reward region of examples. The basic challenge here is that actually producing high-reward examples to train reward model on is hard. If we simply sample responses from the base LLM itself, then it is extremely sample inefficient get the necessary examples (because we are trying to get elements in a low-probability tail). On the other hand, if we use an off-policy procedure-ee e.g.. drawing samples from a stronger I.M,, or producing good examples with extensive thinking or rewrites- -e can get high-reward examples, but naively training reward model on them may learn superficial features instead of eliciting real capabilities (see Appendix D)
To address this challenge,
empirically study rubric-based rewards as solution to this problem. In essence: we get very strong examplar responses by using off-policy generation. Then, we pro- duce reward model using these examples by using another LLM to produce grading rubric for each prompt. Such rubric-based rewards will generalize well off-policy because they are insensitive to irrelevant aspects of the responses by design. The question is then if, and how, rubrics that succeed in capturing the high-reward tail behavior necessary for alignment. We give two principles for achieving this goal. We then produce workflow implementing these ideas and show empirically that it is highly effective for the LLM post-training task.
elicit
Summarizing, the contributions of this paper are:
1. A theoretical characterization of how reward misspecification matters for post-training, concluding that the high-reward region is key,
2. A method for constructing effective reward rubrics using off-policy data, and
3. An empirical study showing the efficacy of the constructed rubries for post-training, and confirming the critical role of misspecificatiio the high-reward region.
2 PRELIMINARIES
Notations. We use to denote a large language model (LLM) and To to denote the reference language model (usually the starting point of RL). Given prompt æ. response y sampled from the conditional distribution .(. z). A reward model r(.,.) is utilized to assess the quality of prompt-response pair. We use x* represent the gold reward model (inaccessible practice) and to represent the proxy reward applied in practice.
Reinforcement fine-tuning (RFT). With prompt set D and a reward model r, the reinforcement fine-tuning optimizes the following objective (Ouyang al. 2022 Bai al. 2022):
max Ep s-(()) [rxx, v)] - (I (T( xx) [)([ [x)]
(1)
where / is hyperparameter to control fine-tuned model's deviation from the reference model, i.e.,
DEL [T [))]) x)] = Be- log
To (y) x))
2

---

Published as a conference paper at ICLR 2026
1.0
0.75
Win Rate
0.8
Win Rate
0.50
0.6
0.25
0.00
KL Divergence
KLL Divergence
Case Correct Top 25% wrong Worst50% wrong
Reversed Top 50% wrong Worsi 90% wrong
(b) Win when different proportions of
(a) Win rate with reward misspecification
top responses are correctly ranked
Figure 2: Theoretical impact of reward model misspecification on performance. (a) Inaccuracy in the high-value region causes performance to collapse. (b) Correctly ranking top responses sufficient for near-optimal performance.
As demonstrated in Rafailov et al.
2023). the solution to ① is
mT (y x))
To ()) exp(r(z,u)/8)
Because RFT relies on proxy rewards in practice, it inevitably suffers from reward over-optimization: the policy exploits inaccuracies in the reward model, achieving high proxy scores while true quality deteriorates. This phenomenon has been well studied in Bradley- Terry reward models trained human preference data (Gao al. 2023). The standard remedy is online RLHF, where fresh human feedback is periodically collected to update the reward model and mitigate over-optimization (Bai et al. 2022) but such approaches are costly and slow.
Reward over-optimization.
Reinforcement learning from rubrics-based reward. Reinforcement learning from rubrics- based reward (RLRR) (Gunjal et al. [025] Viswanathan et al. 2025 Huang et al.] (0225b has emerged promising approach for open-ended tasks. The core idea is associate each prompt with a rubric-a a set of explicit criteria (c;) with corresponding weights (w;) that collectively define high-quality response. For instance, given prompt asking for likely diagnosis from patient's symptoms, the rubric could specify key aspects of a good answer. This might include high-weight criteria for "identifying [correct diagnosis] as the likely diagnosis" and "correctly identifying the condition as a medical emergency," and a low-weight criterion for "mentioning (typical treatment] for treatment" (See Appendix [ for conerete example.)
In this framework, verifier V, typically another LLM, assesses whether given response satisfies each individual criterion. The total reward is then calculated the weighted average of the criteria that the response successfully meets. Formally, the verifier outputs binary score for each criterion, V(x, y, c.) + {0, 1}, and the total reward is:
r(z, v) W,
E,w, E,
RLRR extends Reinforcement Learning with Verifiable Rewards (RLVR) to general tasks where per- formance cannot be easily verified. Compared to RFT using Bradley-Terry reward models, RLRR's explicit criteria make rewards more interpretable and harder to game. However, it's still unclear if, and how, RLRR alleviates reward over-optimization.
HIGH-REWARD REGION ACCURACY IS KEY TO OVERCOMING REWARD OVER-OPTIMIZATION
It's well-known that using misspecified proxy rewards lead to reward over-optimization for rein- forcement post-training. However, the ways in which different misspecification patterns of proxy
3

---

Published as a conference paper at ICLR 2026
rewards influence the performance of the aligned model remain poorly understood. In this sec- tion, we develop theoretical results showing that maintaining high-reward region accuracy is the key determinant of alignment quality.
We introduce a misspecification mapping f from gold to proxy rewards and cast the problem as analyzing how the geometry of f affects performance. More specifically, R -> IR is the mapping from r* to r, i.e., for any z-y pair,
(r, v)) = r(z, v).
To characterize the reward over-optimization phenomenon, we need study the relationship be- tween the utility (expected reward and win rates), and the KI. divergence in (2). They can be sim- plified as follows:
Proposition Define Ri =r" Yo) with Yo Tol- x)) and Fü as its cumulative distribution
function. The RFT solution has:
(i) Expected reward: RewD (-x)()))) (xr, = EAND
FF (RS) CURBIN
(i) Win Rate: EZdD, w-m,( 1 (r* y))] = EenD
B e 1
(iii) KL divergeecc DRL. [N, (yl (T) x)]] = En
log [e [e((R))/8)
B B
To proceed, we assume the current policy's ground-truth reward, Ri is distributed from the standard uniform. This assumption valid since: (i) it matches the reward distribution of best-of-n sampling and the optimal solution which best balance KL divergence and win rate (Gui et al. 2024 Azar al. 2024 Balashankar al., 2024). and (ii) win rate and expected reward matches each other in this case. Under this assumption, we can characterize the utility-KL tradeoff when applying the misspecifed rewards:
Theorem 1. Suppose each R U(0, 1) and f(R)) - Rū. Then holds that:
(i) KL divergence is invariant to
11/8-)e21/ +1
DKL [x T)) x)]
log log(e -1).
e118
(ii Expected reward (or win rate) of is Lo PPa 8(e1/01)
Proof
The explicit formula in Theorem D indicates that misspecification, i.c., the deviation of f from the identity map, in the high-value region of r* has dominantly large effects the utility-KL tradeoff. On one hand, the KL divergence remains invariant to the choice f and is fixed when the penalty parameter & is set. On the other hand, the exponential term imposes increasingly severe penalties on misspecification in the high-reward regime relative to the lew-reward regime. This highlights the criticality of accuracy in the high-reward region for achieving favorable balance between utility and KL divergence.
To verify this, we investigate different, fs and exactly compute the utility-KI. KI. tradeoff curves:
(i) "Correct": identity mapping f(r*) r
(ii) "Reversed":: the reverse mapping f(r* = 1-
(iii) " Top c% wrong": = f'(+* = r"1 (1-s1-e) + (2 -2-c- (ec--r"]((=>11- i.e. the proxy
reward model provides completely reverse rewards for highest quality responses
(iv) "Worst c% wrong": r = f(r*) = (c - r*) L(p + -r*1(r>>)) i.e. the proxy reward model provides completely reverse rewards for worst quality responses

---

Published as a conference paper at ICLR 2026
Proposer
Rubrics
Refized Rubeies
Propossr
Rubriics
Candidate Responses
PRINPT
PRONPT
Verifer
Top
A Pair Responies
Ronkedd Responses
(a) Single-round Improvement
(b) Iterative Improvement
Figure 3: Rubric refinement through response differentiation. (a) Single-round: A proposer LLM
analyzes pair of responses to identify distinguishing features and encodes them new rubric criteria. (b) Iterative: Multiple rounds progressively focus on higher-quality responses, with each iteration filtering to top-scoring candidates before generating new differentiating rubrics.
Figure 2a plots KL divergence versus win rate across misspecification patterns and yields two key observations: (i) when the proxy is inaccurate in the high-reward region, performance may look acceptable at small KL but the win rate collapses as KL grows (this similar to the reward over- optimization behavior in Gao al. (023): ): and (īi) if the proxy correctly ranks just small top proportion of responses (e.g.. 10%) even while misgrading the remaining majority, the win rate rapidly approaches the optimal curve moderate KL. Separately, Figure (6 varies the fraction of correctly ranked top responses and traces the corresponding lower envelope of achievable win rates, showing that this envelope is already near-optimal once sufficiently large top proportion is correctly identified and ordered (e.g.. 40%). Together, we reach our central theoretical findings:
Reward over-optimization primarily arises from the inaccuracy in high-reward regions.
Being able to accurately rank and differentiate high-quality outputs is sufficient for a re- ward model to effectively guide RL.
Algorithm Iterative Rubric Refinement through Progressive Differentiation
1: Input: Pool of candidate responses and initial rubrics
2: Iteration:
For each refinement round:
(a) Score all candidate responses with the current rubrics and get the top responses from the candidate pool as the comparison pair.
(b) Use the proposer LLM to identify distinguishing features between the pair and encode these features by refining the existing rubric set.
3: Output: Final refined rubric
PRINCIPLES FOR CONSTRUCTING RUBRICS
Based on the results of the previous section, we construct a reward model focusing on the high- value region. The problem then is getting training examples that are in this high-reward region. By definition, these are samples that are rare under the base LLM policy! This essentially forces us to use off-policy data to define the reward model. Now, rubric-based rewards have emerged as an approach for using off-policy data to define rewards. The basic idea of rubric-based reward models to explicitly restrict the reward to only care about aspects of the solution that are relevant to its quality, thereby mitigating the side effect of the off-policy data. However, the restrictive nature of the rubrics is a double-edged sword. The same structure that limits the effect of off- off-policyness may also limit their ability to distinguish between solutions that are excellent and those that merely great (they can easily end up in a tie). In this section, we consider how to construct rubrics that are focused on accuracy in the high-reward region.
5

---

Published as a conference paper at ICLR 2026
Table 1: RL experimental results across three domains. The base policy model is Qwen3-8B-Base and the win rate is evaluated against Qwen3-8B. The results show two clear trends: refining rubrics by differentiating two great responses outperforms differentiating two good responses, and differ- entiating among diverse set of great responses further improves performance.
Generalist Domain
Health Domain
Finance Domain
Method
Filtered Set
LMArena
Medical-ol
Finance
Win %
Win
Win %
Score
Win %
Score
Base Policy
5.2
4.1
10.8 25.8
0.1721
5.8
0.1738
SFT
35.9
29.6
0.2999
26.0
0.2218
Initial, Prompt only
31.3
29.7
21.7
0.3004
37.2
0,2693
good Pair
33.5
32.8
22.4
0.2912
39.1
0.2694
great Pair
36.8
33.1
26.5
0.3163
42.2
0.2838
4 great Pairs great & Diverse Pairs
38.7
34.7
31.4
0.3348
48.9
0.2961
39.7
35.1
34.4
0.3513
49.6
0.3018
Refining rubrics to reliably tell apart two already great responses is natural first step toward captur- ing the high-reward tail. To push accuracy further in that tail, we also update rubrics to distinguish among diverse of great responses. We formalize these ideas principles for rubric struction
Principles for Rubric Construction
[Principle 1]
Effective rubric construction requires distinguishing excellent responses from great ones.
[Principle 2] Effective rubric construction requires distinguishing among diverse off off-policy responses.
4.1 METHODOLOGY
To operationalize the above principles, we design an iterative workflow that leverages off-policy responses to refine rubrics.
Refinement-througgh hhrreetttttttuuuugiiiiteeetttttt Diffrrrettttn (RTD). A natural way to make rubric-rewards more dis- criminative is to prompt proposer LLM with a pair of candidate responses and the current rubrics. The proposer analyzes the pair, identifies their distinguishing features, and encodes these distinc- tions as new rubric criteria or refinements of existing ones. We refer to this fundamental refinement step as Refinement-ee Ceatneemnntthruug---ee Maarough--ifferenttattin (RTD).
Iterative workflow for chasing the tail. While a single RTD step sharpens the rubric, repeated application over larger candidate pool yields systematic improvements. Starting with all off off-po responses for a prompt, each iteration scores the candidates under the current rubric, selects the top two responses, and refines the rubric using RTD. This workflow concentrates rubric discovery on the performance frontier, extracting the most informative distinctions from the best available responses with only small number of comparisons (see Algorithm I and Figure 3
4.2 EXPERIMENTAL SETUP
Our experimental goals are twofold. First, examine how leveraging off off-policy responses alle- viate reward over- optimiiimiztionn. Second, we assess the efficacy of these methods enhancing LLM capabilities. Our experiments span three domains: general-purpose, healthcare, and finance. For the

---

Published as a conference paper at ICLR 2026
first goal. to isolate the effect of rubric refinement strategies, adopt the synthetic oracle setting proposed by the seminal work Gao et al. (2023). In this framework, a strong LLM acts as a proxy for the "gold-standard"' preference source. This oracle model serves a dual purpose: it generates the rubrics data used for reward modeling and serves as the final judge for performance evaluation. By unifying the annotation and evaluation sources, this design eliminates confounding factors arising from preference mismatch between the training data and the judge. This allows us to clearly assess how effectively different rubric refinement strategies mitigate reward over-optimization. For the second goal, we evaluate the model performance on domain-specific objective benchmarks when applicable. Regarding the second goal, we specifically evaluate the model on healthcare and finance tasks. These fields offer established objective professional benchmarks, allowing us to rigorously test improvements in model capabilities. We set up the experiments as follows:
Training setup. We employ GPT-4. as the rubric proposer, prompting it to generate the initial rubrics. The training datasets consist of two generalist prompt collections (LMArrea (Chiang et al.) 2024) and a manually filtered set of natural prompts, detailed in Appendix G) and two domain- specific prompt sets: for healthcare, the finance domain, we filtered for 1 1147 high-quality finance prompts in LMArena Team (0025) Each dataset contributes 5000 prompts for training and an additional 1000 prompts for in-domain evaluation, with the exception of the finance dataset, for which we use all available prompts for training and the prompts from the PRBench-Financ, (Akyürekk al. 2025) for the evaluation. The base model for post-training is Breen-8B-BBase ( (Yang et al. 2025). which has instruction-follovin capabilities. We adopt GRPO (Shao et al., 2024) as the RFT algorithm and use a standard set of hyperparameters, detailed in Table For the reward computation, we leverage GPT-4. 1-mini as a rubric verifier and calculate the final reward as the weighted sum of satisfied rubric criteria, normalized by the total weight. All prompts used in the experiments are presented in Appendix
utilize medical-ol -reasoming SFT (Chen et al.] [2024) for
Candidate pool. To validate Principle 1, we compare rubrics refined using (i) candidate pairs from a great model versus (ii) candidate pairs from a good model (Gemini 2.5 Pro and Gemini-2.5- Flash-Lite, respectively (Comanici et al. 20255). To validate P Principle 2, we enlarge the pool by sampling 16 responses per prompt, from a broader set of excellens models, ensuring greater diversity (see Appendix [F] for the full list). This setup allows us to test whether rubric refinement benefits from better and more diverse candidate responses. The supervised fine-tuning (SFT) uses two responses per prompt, sampled from the great model.
Evaluation. We assess performance across two primary dimensions: (1) alignment with oracle preferences, and (2) domain-specific scores on professional benchmarks. For preference evaluation, comparisons against Qwen3-8B, the strong thinking version of our base policy model, on a held-out set of test prompts. The oracle model GPT-4.1 was prompted to act as an impartial evaluator and select the better response (see Appendix D for the detail). To validate the performance gain in professional domains, we additionally evaluate models on HealthBench (Arora jet al., (025) and PRBench (Akyürek et al. 2025). which provide objective metrics grounded in expert-curated rubrics.
we conduct
RESULTS
5.1 RL IMPROVES WITH BETTER AND MORE DIVERSE RESPONSES
We first evaluate downstream RL performance to test whether the proposed principles indeed im- prove rubrics. Table [ shows two clear trends. First, rubrics refined with great pairs outperform those refined with good pairs, validating Principle 1. Second, iterative refinement with multi- plc diverse great pairs yields further gains, validating Principle 2.
Beyond improving average performance, refinement with better and more diverse responses also mitigates reward over-optimizationn Figure 4 shows training dynamics the health and finance domains when RL is run for extended steps. Models trained on initial rubrics, or rubrics refined with only single pair, peak early and then suffer rapid decline in win rate and benchmark scores-ee an indicator of reward oprioiiiiizttion. In contrast, models trained with iteratively re- fined, diverse rubrics sustain higher win rates and benchmark scores for much longer, with over-
7

---

Published as a conference paper at ICLR 2026
30%
0.310
0.325
He Healthheench Seares
25%
Win Rate
20%
0,225
235
0.200
50
100
125
1S0
175
200
25
50
75
100
125
159
375
200
Training Steps
Training Steps
Intia Only
Pair
Pars
(a) Healthcare: Win Rate and HealthBench Scores Different Training Steps
0.24
Win
0.22
20%
0.20
20%
120
30
Training Steps
Inttial Only
Good Pair
CrCwett
Creat
Great Divenne
(b) Finance: Win Rate and PRBench- RRBench-Finance Scores at Different Training Steps
Figure 4: Training dynamics under different rubric construction strategies separate prolonged training. The figures show the evolution of the Win Rate and respective benchmark scores in the healthcare (a) and finance (b) domains.
optimization not appearing until late stages. This pattern indicates that refining rubrics with great and diverse responses corrects inaccuracies in the high-reward region, thereby delaying the onset of over-optimizationn Together, these results confirm our central hypothesis that rubrics can be con- structed to mitigate reward over-optimization.
5.2 REWARD MODEL ACCURACY IMPROVES THE HIGH-REWARD TAIL
Our theoretical analysis (Section 3) suggests that accuracy in the high-reward tail is the critical factor for downstream RL performance. To understand why refinement with better and more diverse re- sponses helps, we evaluate the agreement between rubric-based rewards and the ground-truth judge, separately on the high- and low-reward regions.
As shown in Table incorporating any candidate responses through refinement improves rubric accuracy compared the prompt-only baseline. More importantly, rubrics refined with great pairs largely improve accuracy in the high-reward region, while good pairs improve accuracy more than great pairs in the low low-reward region. Iterative refinement with great pairs pushes the accuracy in the high-reward region even further, mirroring the RL improvements Table This confirms that both principles work by sharpening reward model accuracy where matters most: the high-value tail.
8

---

Published as a conference paper at ICLR 2026
Table 2: Accuracy rubric-based scoring in predicting ground-truth model preferences was evalu- ated on 1000 random prompts from the training set. Response pairs in the high-reward region were sampled from Qwen3-8B, and response pairs in the low-reward region were sampled from Qwen3- 8B-Base. Rubric preferences were determined by majority vote from five independent gradings, with ties counted as incorrect. Results how refining with stronger and more diverse responses improves high-reward accuracy.
Initial Only
Good Pair
Great Pair
4 Great Pairs 4 Great & Diverse Pairs
High-reward Low-reward
40.3%
42.2% 67.9%
45.8% 66.7%
49.2%
47.9%
66.2%
68.9%
69.8%
5.3 REFINEMENTS FROM BETTER RESPONSES ARE MORE SOPHISTICATED
Finally, we analyze how refinements differ when using good versus great candidate responses. To understand how stronger candidate responses lead to better rubrics, we analyzed the types of re- finements made when using different quality levels of candidate pairs. We prompted LLM to compare initial and refined rubrics, and categorized the improvements into semantic clusters (see details in Appendix H).
Table 3 shows the distribution of refinement types on the health domain. Both qualities contribute, but the patterns diverge: good responses often drive basic corrections, such adding penalties for obvious mistakes or broadening overly restrictive criteria; by contrast, great responses more often drive sophisticated refinements, such as breaking down complex criteria into sub-components or enhancing verification standards.
In the example from Appendix [ for a medical prompt about a patient with serious symptoms, two initially tied great responses are distinguished by adding the criterion: "The response mentions that urgent imaging (e-g., contrast-enhaneee This refinement, from the "Enhancing verification, validation, and evidence standards" cluster, man- dates a critical, verifiable clinical action, and only one of the responses satisfies. Such qualitative results confirm finding that comparing great responses provides the nuanced distinctions needed to identify excellent outputs, thereby sharpening accuracy in the high-reward tail.
CT MRI/MRV) required to confirm the diagnosis."
Table 3: Distribution of rubric refinement types when using great (Gemini 2.5 Pro) versus good (Gemini 2.5 Flash Lite) candidate pairs, in the healthcare domain. Rows with significant differences (255% for model) highlighted: blue indicates great dominance, red indicates good dom- inance. Bold percentages show the dominant model.
Refinement Type
Proportion
Great Good
Mandating explicit statements, justifications, or declarations Shifting focus from superficial to substantive qualities
16.7%
52.6%
47.4%
11.7%
48.2%
51.8%
Adjusting scoring weights, granularity. or mochanisms Breaking down complex criteria into sub- components Introducing penalties, prohibitions, or negative scoring Replacing vague language with specific requirements Adding roquirements for comparing alternttives
8.9%
48.5% vs 51.5%
7.2%
55.9%
44.1%
6.6%
43.5% vs 56.5%
6.2%
54.7% vs 45.3%
5.9%
49.0% vs 51.0%
Broadening criteria accept multiple approaches Adding conditional or context-dopenden rules Streamlining by removing redundancy
55.6%
4.5%
11.4% 48 6%
4.4%
41.6% vs 58.4%
Adding timing, sequencing. process flow criteria Mandating precise language or technical accuracy
3.5%
54.1% 45.9%
3.3%
50.2% vs. 49.8%
Requiring cansal explanations mechanistic understanding Enhancing verification, validation, and evidence standards Mandating specific structure lormatting
3.3%
51.8% 48.25
2,3%
55.0% % vs 45-45.0 45.0%
2.1%
Requiring explicit justification for decisions Delining explicit scope, boundaries, or constraints Incorporating risk analysis safety constraints Requiring specific, actionable recommendations Correcting aligning with intended standards
1.9%
50.3% 49.7%
188%
58.9 % vs 41.1%
1.8%
55.2% 44.8%.
1.0%
$5.5 %
vs. 44.5% 5%
0.9%
32.8%
67.2%
Assessing communication quality or tone
0.5%
43.8%
vs 56.2%

---

Published as a conference paper at ICLR 2026
6 RELATED WORK
Reward over-optimization. Gao et al. (2023) highlighted the issue of reward over-optimization for both best-of-n sampling and reinforcement learning when using preference-based reward mod- els. Although this phenomenon has since been repeatedly observed in empirical studies (Bai et al.,
2022 Moskovitz et al., 2023; Perez al. 2023] Gui al. 2024 Wang al. 2024), its theoret- ical underpinnings remain limited. Existing analyses typically relate the performance degradation caused by proxy reward global statistics describing how far the proxy deviates from the true reward (Huang et al., 2025a [Moueh] 2024). In contrast, our work provides a sharper perspective: what truly governs performance is the fidelity of the proxy reward in the high-value region, where high-quality responses concentrate.
Rubrics reward. RL from rubrics reward (RLRR) has proven to be an effective method in spe- cialized domains like science and health (Gunjal al. 2025), general instruction-follovie
(Huang et al. 20256 Viswanathan et al., 2025), and for enhancing agentic ability (Team et al. (025). with implementations using both online and offline RL. The idea of rubrics is also utilized in generative reward models (GRMs), wherein a reward model is prompted to first generate rubrics and then use them to evaluate response (Liu al. 2025b; Chen et al. 2025). This approach enables inference time scaling of reward modeling and improves explainability. However, generating rubrics on the fly is computationally inefficient and unsuitable for large-scale training.
DISCUSSION
In this paper, we investigate rubric-based reward modeling for LLM post-training. We begin by analyzing the central weakness of reinforcement fine-tuning, reward over-optimization, and theo- retically trace it to misspecification of the proxy reward the high-reward tail. A comprehensive empirical study highlights rubric-based rewards an effective remedy. We further demonstrate that carefully designed rubrics, which distinguish among great, diverse off-policy responses, lead to consistently strong fine-tuning performance.
Off-pollcy responses for Bradley-Terry reward model training might generalize, but is sample inefficient. While we find medium amount off-policy responses (n 5000, in addition to the same number of on-policy responses) do not help Bradley-Tery reward model guide the current policy (see Appendix D we note that other work successfully train BT reward model with off- policy samples, but with much larger scale-using up to 20 million high quality samples ((Liu) jet al., 2025a; Cui et al. (023)). Indeed, Bradley-Terry reward model's generalizability scales with the number, and diversity of training samples. However, it's not always easy to find large-scale data for many specialized domains, such as healthcare. contrast, rubric-basee reward can easily encode generalizable principles from limited amount of data.
Weighted average of rubric score is not optimal. To specifically analyze the impact of rubric quality, we deliberately the most simple method of score aggregation, by taking a weighted av- erage of scores from the satisfied criteria. Prior work has explored diverse approaches, including im- plicit aggregation by verifier model (Gunjal et al.) (025), sophisticated frameworks capture non- linear dependencies (Huang et al.) 2025b), weighted averages of continuous scores (Viswanathan al. 2025), and moddel-based self self-critique that weighs criteria against internal priors (Team al., 2025 We acknowledge that aggregation is central component of an optimal rubric reward system and leave it for future work.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Afra Feyza Akyürek, Advait Gosai, Chen Bo Calvin Zhang, Vipul Gupta, Jaehwan Jeong, Anisha Gunjal, Tahseen Rabbani, Maria Mazzone, David Randolph, Mohammad Mahmoudi Meymand, et al. Prbench: Large-scale expert rubrics for evaluating high-stakes professional reasoning. arXiv preprint arXiv:2511.11502 2025.
Rahul K Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quifonero-Candela Foivos Tsippouraas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, et al. Health- bench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775. 2025.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447 4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05800 2022.
Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Eisenstein, Michael Collins, Adrian Hut- ter, Jong Lee, Chirag Nagpal, Flavien Prost, Aradhana Sinha, et al. Infalign: Inference-aware language model alignment. arXiv preprint drXiv:2412.19992 2024.
Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang. Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-ol, towards medical complex reasoning with llms, 2024. URL. abs https://arxiv.cor btrrg/bbs/4113 22120 18925
Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, et al. Rm-rl: Reward modeling as reasoning. arXiv preprint arXiv:2505.22887, 2025.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating Ilms by human preference. In Forty-first International Conference on Machine Learning, 2024.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.0626h 2025.
Gangu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with sealed ai feedback. arXiv preprint arXiv:23 10.01377. 2023.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 18835-10666, PMLR, 2023.
Lin Gui, Cristina Gårbacea, and Victor Veitch. Bonbon alignment for large language models and the sweetness best-of-n sampling. arXiv preprint arXiv:2406.00822 2024.
Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains, 2025. URL
htttss://arxe wwtp://ogg//aapaeeeeo
17746
Audrey Huang. Adam Block, Qinghua Liu, Nan Jiang, Dylan J Foster, and Akshay Krishnamurthy. Is best-of-n the best of them? coverage, scaling, and optimality inference-ting alignment. arXiv preprint arXiv:2503.2 1878, 2025a.
11

---

Published as a conference paper at ICLR 2026
Zenan Huang, Yihong Zhuang, Guoshan Lu, Zeyu Qin, Haokai Xu, Tianyu Zhao, Ru Peng, Jiaqi Hu, Zhanming Shen, Xiaomeng Hu, et al. Reinforcement learning with rubric anchors. arXiv preprint arXiv:2508.127990 2025b.
Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, et al. Skywork-reward--2. Scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.01352, 2025a.
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. arXiv preprint arx11:2504.024995, 2025b.
LMArena Team. A deep dive into recent arena data. https://news. http://awww..aaaaaaaaaa
Lmarena-ai/arenaa-aaaaa heapraameeemmmp-ppeefeeeeee-ee- httpss//hhgggggaccccc///aaaaeee
Shendatta-uuly22253 2025. Dataset: https:/
Ted Moskovitz, Aaditya K Singh, DJ Strouse. Tuomas Sandholm, Ruslan Salakhutdinov, Anca D Dragan, and Stephen McAleer. Confronting reward model overoptimization with constrained rlhf. arXiv preprint arXiv:2310.04373, 2023.
Youssef Mroueh, Information theoretic guarantees for policy alignment in large language models. arXiv preprint arXhv:2-40.008883 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 17730-27744 Curran Associates, Inc., 2022. URL bttpsss/ppooceeddngg..eell neuri cc wwwpppppppppp.eeeeeecceceeeee..e http:/paapprreeesepeeepe fwwes/paper////// htte://..eeeeeeeeeeaa11111331333 -orepa .df
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-writte evaluations. In Findings of the association for computational lin- guistics: ACL 2023, pp. 13387-13434, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea
Finn. Direct preference optimization: Your language model secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:
//openzeview. net http://oooinn.aaPPSSS..... d==HusIXJaas
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.033000, 2024.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, al. Kimi K2: Open agentic intelligence. arXiv preprint arXiv:2507.20534 2025.
Vijay Viswanathan, Yanchao Sun, Shuang Ma, Xiang Kong, Meng Cao, Graham Neubig, and Tong- shuang Wu. Checklists are better than reward models for aligning language models. arXiv preprint arXiv:2507.18624 2025.
Zihao Wang, Chirag Nagpal, Jonathan Berant, Jacob Eisenstein, Alex D' Amour, Sanmi Koyejo, and Victor Veitch. Transforming and combining rewards for aligning large language models, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang. Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
12

---

Published as a conference paper at ICLR 2026
USAGE OF LARGE LANGUAGE MODELS
In this work, besides running LLMs in experiments, we use LLMs for the following purposes:
1. Aid or Polish Writing (Gemini 2.5 Pro, ChatGPT 4/5)
2. Literature Retrieval and Discovery (e.g., finding related work) (Gemini 2.5 Pro Deep Re- search, ChatGPT Deep Research)
3. Assisting Code Writing and Debugging (Css41e-Ops-411, GPT-5)
We fully understand the responsibility of using LLMs academic research. We carefully monitor any potential problems, such as plagiarism or scientific misconduct (e.g., fabrication of facts) when using LLMs. We make sure these problems do not occur the paper.
A THEORETICAL RESULTS
Theorem 1. Suppose each Ri UO0, 1) and f(R)) d
Rj- Then it holds that
(i) KL divergence is invariant to
DeL ( To)l x)] - -1/8-1)e"/
+1
log 8 log(e" 113 1).
e21B
(ii) Expected reward (or win rate) of is So 1 (w)e=//dd 1122/011)
Proofy
Proof. First, we compute the KI. divergence. When f(R)) U0, 1), by Proposition divergence is
the KL.
E [[(R)) el (RBVD /B
DEL (v [ 2)] BEnD
log E c [
[e
So //e du
log (J" curPdty)
(1/3- - 1)81/8 +1 log 3(6 11)100-1)-- -1)
B.-D
B So ew/9
11/8
Then, compute the expected reward: denote Tø f(R)).
E
Rī e (RR)/
EznD, (+(x) (r (z, v)] = ELnD
(R)))/
s (T TT110
So f (w)e
E
E. [e
So en/Pdu
J f- (uuee//du
So f
4/)
So e//Pdu
B
Since (16)) when R5 ~ UO, 1), the win rate is the expected reward. Then the theorem follows.
13

---

Published as a conference paper at ICLR 2026
B PROMPTS USED FOR EXPERIMENTS
Prompt for Constructing Initial Rubrics
You're a skilled judge evaluating the quality of LLM responses to a user prompt. Your first task is create comprehensive rubric for grading these responses across multiple dimensions.
Given user prompt, generate a list of binary (yes/no) criteria. These criteria should assess how well the LLM answered the prompt. Only write rubries you are confident about.
Here are tips writing good rubrics:
i. MECE:
-Mutually Exclusive, Collectively Exhaustive
ii Completeness:
Connsidrr all the elements you would want to include to create a perfect response and put them into the rubric. This means including not only the facts and statements directly requested by the prompt, but also the supporting details that provide justification, reasoning, and logic for your response. Each of these elements should have criterion because each criterion helps to develop the answer to the question from a slightly different angle.
iii. No overlapping: the same eтror from a model shouldn't 't be punished multiple times.
iv. Diversity:
-The rubric items should include variable types of information. If all criteria are like "the response mentions A", "the response mentions B", then this not a good rubric.
v: How many rubric items for each prompt There is no golden standard, and the desired number of rubrics varies by accounts and task types.
Wriite rubrics that cover all aspects of ideal response.
vi: How many rubric items fail
A good rule of thumb is that the model fails on 50% of rubrics items
vii: Atomicity Non-stacked
Each rubric criterion should evaluate exactly one distinct aspect. Avoid bundling mul-
tiple criteria into single rubric. Most stacked criteria with the word "and" can be broken up into multiple pieces.
X Response identifies George Washington the first U.S. president and mentions he served two terms.
Response identifies George Washington the first U.S. president. Response mentions that George Washington served two terms.
viii: Specificity
Criteria should be binary (true or false) and objective.
Avoid vague descriptions (e.g., "the response must be accurate" is vague).
Example: "TThe response should list exactly three examples."
14

---

Published as a conference paper at ICLR 2026
ix: Self-contained
Each criterion should contain all the information needed to evaluate a response, e.g.
X Mentions the capital city of Canada.
Mentions the capital city of Canada is Ottawa.
Criterion should be verifiable without requiring external search.
X Response names any of the Nobel Prize winners in Physics in 2023 Response names any of the following Nobel Prize winners in Physics in 2023: Pierre Ago Agostini, Ferenc Krausz, or Anne L'Huillier.
xi. The binary criteria should be phrased so that yes means the model response good and means the model response is bad.
Finally, we want to assign different weight for each question. Give weight on scale of (least important) to (most important) for each question based on
the question's 's alignment with user demand if user would be frustrated if the answer no; 1 if user would not be bothered at all if the answer is no)
the question's 's importance in terms of determining quality/correcttess (3 the response would be completely incorrect if the answer is no; if an extreme edge case would be missed and the overall quality won't 't't be affected if the answer no)
Here the user prompt for which we want to generate a rubric:
PROMPT:
[prompt)
Return ONLY the JSON array of the rubrics, no other text. For example:
(("eriterion": "Does the response provide list of songs?", "weight": } (("criterion"; "The response explicitly state it is listing French romantic songs.", ", "weight": 21}
Note: Local IDs will be automatically assigned to each criterion (cl, c2, c3, etc.), so don't include IDs into outputed criterion.
Prompt for Improving Rubrics
You're skilled judge assessing the quality of LLM responses to a user prompt. The current nubric isn't good enough to effectively differentiate between high-quality responses.
Your goal is improve the current rurbies to address this (adding creteria, rewriting. decomposing, and deleting the current creteria). The updated rubric must be comprehensive and consistently applicable for grading LLM responses. These criteria should specifically assess how well the LLM answered the given prompt. Only write rubrics you are confident about.
Here are tips writing good rubrics:
15

---

Published as a conference paper at ICLR 2026
i. MECE:
-Mutually Exclusive, Collectively Exhaustive
ii. Completeness:
Consider all the elements you would want include create a perfect response and them into the rubric. This means including not only the facts and statements directly requested by the prompt, but also the supporting details that provide justification, reasoning, and logic for your response. Each of these elements should have criterion because each criterion helps to develop the answer to the question from a slightly different angle.
iii. No overlapping:
the same error from model shouldn't it be punished multiple times.
iv. Diversity:
The rubric items should include variable types of information. If all criteria are like "the response mentions A", "the response mentions B", then this is not a good rubric.
How many rubric items for each prompt
There no golden standard, and the desired number rubrics varies by accounts and task types.
-Write rubrics that cover all aspects of an ideal response.
vi: How many rubric items to fail
A good rule of thumb is that the model fails 50% of rubrics items
vii: Atomicity Non-stacked
Each rubric criterion should evaluate exactly one distinct aspect. Avoid bundling mul-
tiple criteria into single rubric. Most stacked criteria with the word "and" can be broken up into multiple pieces.
Response identifies George Washington as the first U.S. president and mentions he served two terms.
Response identifies George Washington as the first U.S. president. Response mentions that George Washington served two terms.
viii: Specificity
Criteria should be binary (true or false) and objective.
Avoid vague deseriptions (e.g., "the response must accurate" is vague). Example: "The response should list exactly three examples."
ix: Self-contained
Each criterion should contain all the information needed to evaluate a response, e.g.
x Mentions the capital city of Canada.
Mentions the capital city of Canada is Ottawa.
x: Criterion should be verifiable without requiring external search. X Response names any of the Nobel Prize winners in Physics in 2023 Response names any of the following Nobel Prize winners in Physics in 2023: Pierre Agostini, Ferenc Krausz, or Anne L'Huillier.
xi. The binary criteria should be phrased so that yes means the model response good and no means the model response is bad.
16

---

Published as a conference paper at ICLR 2026
Finally, want to assign different weight for each criterion. Give weight scale of (least important) to (most important) for each question based on the question's s alignment with demand if user would be frustrated if the answer is no; 1 user would not be bothered at all the answer is no)
the question's importance in terms of determining quality/correctness (3 the response would be completely incorrect if the answer is no; if an extreme edge case would be missed and the overall quality won't be affected if the answer no)
Here the user prompt for which we want to improve the rubric:
PROMPT:
{prompt)
The existing rubrics we using is: {rubrics}
The two reference responses are:
Reponse 1: {response 1
Reponse 2: [response2]
Return ONLY the JSON array of the full rubries, other text. For example:
(""rriterion": "Does the response provide specific release years for each song?", "weight"; 2)}, (("criterion": "The response includes artist names for each song mentioned", "weight": 1}}
]
Local IDs will be automatically assigned each criterion, so don't include IDs in your output.
Note:
Prompt for Scoring Responses
You are skilled judge who will assessing the quality of LLM responses to a user prompt.
Given a user prompt, LLM response, and a rubric, your task is evalauting the performance of the model response by seeing whether or not it meets the rubric dimension.
Answer the each of the given rubric dimension in either "yes" or "no". Do output any response other than "yes" or "no".
Keep in mind that you will be grading industry-leading LLMs. Make sure to have high expectation for grading the responses.
17

---

Published as a conference paper at ICLR 2026
Make your evaluation is as objective and consistent as it could be. By consistent mean that different evaluator's assessment of the task should agree with yours.
Think carefully before you make the decision. After you make the decision, explicitly output which dimension receives "yes" and which dimension receives "no".
Input:
* PROMPT: {prompt}
RESPONSE: {response}
RUBBIC: }
Return ONLY the JSON array, no other text. For example:
(("cl": "yes", "c2"; "no", "c3": "yes")
18

---

Published as a conference paper at ICLR 2026
C HYPERPARAMETER
The GRPO hyperparameters used for the RLRR results Table are listed in Table The finance domain is an exception due to the limited amount of available data, and its hyperparameters presented separately T Figure uses a longer training run than in table [ resulting in some performance variations due to different warmup steps.
Table 4: GRPO Hyperparameter Configuration
Hyperparameter
Value
Rollouts per Prompt
16
Gradient Accumulation Steps
2
Per-Device Train Batch Size
Warmup Ratio
0.1
KL Coefficient
0.01
Learning Rate
1.0
10-5
Learning Rate Scheduler
Constant with Warmup
Maximum Sequence Length
3584
Training Epochs
2
Table 5: GRPO Hyperparameter Configuration in Finance Domain
Hyperparameter
Value
Rollouts per Prompt
16
Gradient Accumulation Steps
2
Per-Device Train Batch Size
Warmup Ratio
0.1
KL Coefficient
0.01
Learning Rate
1.0
10-5
Learning Rate Scheduler
Constant with Warmup
Maximum Sequence Length
3584
Training Steps
320
D EMPIRICAL RESULTS ON RLHF
We finetune Bradley-Terry Reward model various responses, with preference generated by 1PT-4.1, the same model as the judge model for evaluation. For each of the prompt in the training set, we generated a pair of responses temperature 1.0 using the base policy model Qwen3-8B- Base (on-policy) or Gemini-2.5-Pro (off-policy). Preferences were labeled using GPT-4.1, 1, the same model used for final evaluation. This preference data was then used to train reward model based on Llama-3. 1--8--IIntuuu Instruct, with byperparameters specified in Table 7 Finally, this reward model was used for GRPO training, following the configuration in Table
We find that using on-policy responses is baseline that can't be easily improved upon:
1. Training on off-policy, great responses deteriorates the performance
2. Adding both off offlcy and on-policy responses only helps with win rates but not helps with healthbench. This suggests that the off-policy samples only help the reward model encode superficial features (that can game LLM-judge) instead of true capabilities as measured by more objective metrics.
This experiment shows the difficulty of improving Bradley-Terry models with off-policy responses.
19

---

Published as a conference paper at ICLR 2026
Table Win-rates and HealthBench scores for the Health domain.
Method
Win-Rate
HealthBench
Reward Model (on-policy)
26.8%
0.3036
Reward Model (off-policy, great)
22.4%
0.2798
Reward Model (on-policy off off-policy)
30.7%
0.3032
SFT
25.8%
0.3094
Initial, Prompt only
21.7%
0.3004
1 good Pair
22.4%
0.2912
great Pair
26.5 %
0.3163
4 great Pairs
31.4%
0.3348
4 great & Diverse Pairs
34.4%
0.3513
Table 7: Reward Model Hyperparameter Configuration
Hyperparameter
Value
Learning Rate
1.0 x 10-5
Per-Device Train Batch Size
4
Gradient Accumulation Steps
Training Epochs
10
Maximum Sequence Length
8192
Warmup Ratio
0.1
Learning Rate Scheduler
Cosine
E LLM JUDGE FOR EVALUATION
We use the same judge model as the rubrics proposer (GPT-4.1). This by design: our primary goal is to how best incorporate additional responses into the rubric construction process. By using the same powerful model for both proposing rubrics and evaluating final outputs, we isolate the quality of the candidate responses as the key experimental variable and eliminate potential confounding issues that could arise from disagreements between a proposer and judge.
We use minimal judge prompt compare two responses:
LLM Judge Prompt
You are skilled judge who will be assessing the quality of LLM responses a user prompt.
Avoid any position biases and ensure that the order which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation.
Here is the user prompt:
PROMPT: {prompt}
The two responses are:
Response 1: (responsel)
Response 2: {response2)
Which reponse would you prefer? Enclose your final answer or in |boxedf[...]].
20

---

Published as a conference paper at ICLR 2026
To reduce the position bias, randomly flipped two responses.
F FRONTIER MODELS USED TO CREATE CANDIDATE RESPONSES
The 16 frontier models used to generate candidate responses are:
Gemini-2.5-PB
Gemini-2.5-Flash
GPT-5
GPT-4.1
GPT-44--0224005-3
01-2024-12-17
04-mini
4
2omude-3-7-Somne-222222
Deepseek-
Deepseck-RR R1
Kimi-K2 -2-truettt
GLM-4.5
Mistral-Medium-Laee
G PRINCIPLES OF SELECTING PROMPTS
We manually curated a prompt dataset according to specific set of criteria to ensure quality and suitability for rubrics training.
Prompts have clear user intent.
Prompts mutinnoda/ssacchhiiiaaiTTTAA
Prompts are not too simple.
Prompts are generalist friendly (nothing technical that requires expert knowledge).
Prompts are not open-ended creative-writing
Prompts are designed such that there is an objectively better response ("Tell me a good bedtime story" can have multiple good responses)
H PATTERN DETECTION ON RUBRIC REFINEMENTS
Table we show clusters of various rubric-refinements. To get this result, we first prompt GPT- 4.1 to describe the differences between refined rubrics, and the initial rubrics (for refinement with one- tee--ood-puee -air,, and see--grrt--prr)) The differences are listed as atomic items describing different aspects of improvement. We then run clustering algorithm one- caeegood--pie and one-great-par (they have almost the same number of atomic differences) to get interpretable clusters, and report the proportions from one-good good-pair versus and mm--great-pain
the atomic differences from both
21

---

Published as a conference paper at ICLR 2026
I EXAMPLES OF RUBRICS AND RUBRIC REFINEMENTS
In this example, the initial rubric cannot differentiate between the two grear responses (both from Gemini-2.5-Proo, This is because it primarily handles basic criteria (e.g. the diagnosis correct), which both responses satisfy. The refinement process resolves the tie by adding a new criterion c7 e7-requin the mention of an imaging test confirm the diagnosis. Such refinement one example for the cluster Enhancing verification, validation, and evidence standards identified in Table 3
Rubric Refinement Example
Prompt
What is the likely diagnosis for young giri with history of repeated poin over the medial conthau and chromic use of decongestauur, who now presents with intense chilis, rigors, diplopia on lateral gaze, congested optic dise on examination?
Response
Response 2
Most likely diagnosis: Cavernous sinus thrombusis (CST).
Most likely diagnosis: Cavernous simus thrombosis (CST), secondary to dacryocystiirs/orhintl cellulitis. This is life- threatening emergency.
This is life-th life-threatening medical emergency.
CST
Clinical reasoning
Medial canthus decongestants sinusitis
thalmic communicate valveloss oph- chills/rigors thrombophlebiti..
Medial canthus pain localizes to lacrimal sac; chronic congession/decoogessttts imply lucrinaal dysfumction and recurrent dacryocystilis. Valveless facial/ophthalmicc veins penmit retro grade spread cavernuss sinus.
septicemia septic
Diplopia on lateral gaze - abducens nerve (CN VT) involvement.
CST manifestations
Congested optic dise + impaired venous outflow increased ICP.
Sepsis (chills, rigons).
CN VI palsy causing diplopia lateral gaze. Papilledemn from impaired drainage
Differential: osbital cellulitis; meningitisbrain abscess.
Inmediate management: urgent contrast MRI/MV;
creased ICP.
high dove IV antibiotics; consider anlicoagulation; drain source indicated.
Urgency treatment theurosrher emergency; high- dosc IV antibioties essential.
Rubric Scores (Tnitial)
Criterion (Initial)
Weight
(0/1)
(0/1)
c1: Identifies CST most likely diagnosis
medical emergency
<3: Links medial canthus pain + decongestants to sinusitis
c4: Diplopia due CN involvement
<5: Papilledemm from imguiiedd vemous drainge//CPP
06: Chills/rigrrs systemic aafetiioneacteremia
<:: Includes medical disclaimer seek care
<R: Mentions orhiral cellulitis differential high
untibiotics
Weighted total (Iiiiia):: RI = 18/20, 20 R2 18/20
Rubric Scores (Refined)
Criterion (Retined)
Weight
RI ((V1)
R2 (OV1)
<l: Identilies CST most likely diagnosis
<2: Explicitly states CST medical emergency
63: Links medial canthus pain decongestants to sinnsitiv/daerroeestii
<4: Diplapia due
involvement
c5: Papilledema from imgaized venous drainage/CCP
Scpsis secondary CST (chills/tioors)
c7: Urgent imaging (contrast CT or MRI/MRV) required to confirm diagnosis
<8: High-dose TV antibiorics initial mainstay
<9: Medical disclaimer seek immnediate cure
c10: Mentions orbital cellulitis differential
IV, VI, V2) may affected c12: Avoids incorrect primary diagnosis
Notes other
Weighted Refined);
25/27, 22/27
22

---

Published as a conference paper at ICLR 2026
J RUBRICS CATEGORIZATION
We find the constructed rubrics encode diverse and concrete criteria instead of superficial stylistic features. To see this, we systematically study rubrics for the health domain constructed with the workflow using 4 pairs from Great & Diverse candidate responses. Since each rubric criterion assesses certain capability of policy models, we apply hierarchical K-means clustering to the targeted capabilities of each criterion. This clustering analysis yields 20 types of capabilities, with their distribution presented in Figure 5 For each type, we illustrate its meaning with an example in the following box. We also provide the full clustering results with all examples in the supplement material.
Turgding Model Casatily Trpet
Nutying, a-rers uileorddi aediatcionnnn
jutt
autemstiilly dirreenng atoraak posshes bared mridero. 840%
Systmmace appicatio pricc tn
1.1%
fedioal
Staaiees 3,29
Nosvere infomaatio Starig -REAEEE
and fantlann fini/madi
Cormunceing with darty scaping sudiaco caal
wc 252h
1.22%
Systema
ditin testin Mame haae inseliaaee 3.5%
Gur sning casa ugo cance canaaaal qurtcaiie rulyii.
6.6%
Figure The Distribution of Rubrics Targeting Each Type of Model Capabilities
Categorization of Target Capabilities of Rubrics
Target Capability Type 1
Analyzing causes, predicting consequences, and explaining rationale
Example:
Prompt: A 62-year-old man presents with inability to tan and mild fatigue. Laboratory tests indicate mild anemia with hemoglobin reticulocytes. Given these findings and the peripheral blood smear results suggesting hemolytic anemia, what specific patient history or characteristic is most consistent with the mechanical destruction of red blood cells the underlying cause?
g/dL, decreased haptoglobin, and 3%
Rubric: The response explains that decreased haptoglobin is due to binding to free hemoglobin released during red blood cell lysis.
Target Model Capability: Connecting specific observations or data points their corresponding causes, effects, or implications within larger system.
Target Capability Type 2
Systematic analysis and comparative evaluation for risk assessment and justified decision- making in critical situations.
Example:
Prompt: patient presents in a stuporous condition, with a history of persistent sadness, suicidal attempts, and a lack of cating and sleeping according to the patient's parents. What the recommended treatment in this case?
Rubric:
The response recommends inpatient hospitalization for safety, evaluation, and stabilization (not outpatient or home-based care).
Target Model Capability: Identifying critical conditions requiring immediate and decisive intervention based on severity assessment.
23

---

Published as a conference paper at ICLR 2026
Target Capability Type 3:
Developing comprehensive, actionable, and risk-aware strategic plans.
Example:
Prompt: What is the ideal management for a 5-year-old boy with retinoblastoma involving the entire right eyeball and 2-3 small lesions in the periphery of the other eye?
Rubric: The response specifies the timing of prosthetic eye fitting after enucleation (e.g.. 4-6 weeks post-surgery).
Target Capability: Detailing a post-intervention protocol by specifying follow-on diagnos- tic assessments and subsequent restorative procedures with timelines.
Target Capability Type 4:
Structured and logically clear presentation.
Example:
Prompt:
What is the most likely diagnosis for 3-year-old child who presents with eczematous dermatitis on extensor surfaces and has mother with history of bronchial asthma?
The response organized logically, with clear separation between diagnosis, reasoning, differential diagnoses, and any disclaimers.
Rubric:
Target Capability: Structuring the response logically by partitioning distinct conceptual elements into clearly delineated sections.
Target Capability Type 5:
Analyzing components, relationships, and functions within complex systems.
Example:
Prompt: What the most probable diagnosis for a 6-year-old boy who has been experi- encing headaches and peripheral vision loss for four months, with CT scan showing suprasellar mass with calcification?
Rubric: The response links peripheral vision loss compression of the optic chiasm by the mass.
Target Capability: Justifying conclusion by explicitly linking individual pieces of evidence to their respective supporting roles in the final determination.
Target Capability Type 6:
Executing sequential calculations and quantitative procedures involving formulas, unit conversions, and operational precision.
Example:
Prompt: A 1-year-old child weighing 6 kg is suffering from Acute Gastroenteritis, showing signs sunken eyes and a skin pinch test indicating rapid fluid replenishment. Based on these symptoms, what volume and rate of Ringer's Lactate infusion would you administer over the first six hours?
Rubric: The response provides the correct infusion rates for each phase: 180 mL/hour for the first hour and 84 mL/hour for the next five hours.
Target Capability: Executing precise, multi-step quantitative calculation based on given inputs and established formulas to derive phased implementation plan.
Target Capability Type 7:
Resolving inconsistencies, ambiguities, and anomalies data.
Example:
Prompt: A patient with head injury admitted to the intensive care unit showing signs of raised intracranial pressure. He is placed on ventilator and given intravenous fluids and diuretics. After 24 hours, the patient's urine output is 3.5 liters, serum sodium level 156
24

---

Published as a conference paper at ICLR 2026
mEq/1, and urine osmolality is 316 mOsm//g. What is the most likely cause of these clinical findings?
The response identifies the urine osmolality of 316 mOsm/kg inappropriately low for the degree hypernatremia (i.e., urine should be more concentrated in this context).
Rubrie:
Target Capability: Evaluating the relationship between multiple variables to identify paradoxical inconsistent patterns relative expected system behavior.
Target Capability Type 8:
Verifying factual accuracy, correcting misinformation, and attributing information to authoritative sources.
Example:
Prompt: According to the latest resuscitation guidelines, for how long must umbilical cord clamping be delayed in preterm infants?
Rubric:
The response identifies at least one authoritative organization issuing the guideline AHA, AHA, ILCOR, ACOG, WHO, ERC, NRP).
Target Capability: Attribute the factual knowledge to the authoritative sources.
Target Capability Type 9:
Synthesizing. analyzing, and contextualizing evidence to derive well-supported conclusions.
Example:
Prompt: A 33-ycar-old woman is brought to the emergency department 15 minutes after being stabbed in the chest with screwdriver. Given her vital signs of pulse 110mmin, respirations 22/min, and blood pressure 90/65 mm Hg, along with the presence of a 5-cm deep stab wound at the upper border of the 8th rib in the left midaxillary line, which anatomical structure in her chest is most likely to be injured?
The response provides clear, logical synthesis connecting wound location, anatomical relationships, and clinical findings to justify the conclusion.
Rubric:
Target Capability: Synthesizing multiple distinct lines of evidence into coherent, logical argument justify final conclusion.
Target Capability Type 10:
Analyzing, defining, and differentiating abstract concepts and ideas.
Example:
Prompt: 68-year-old woman with elevated serum calcium, high parathyroid hormone, low phosphorus, and a history of kidney stones presents with fatigue, constipation, diffuse bone pain, and 24-hour urine calcium level that is elevated. Given these clinical and laboratory findings, what radiologie finding on hand X-ray would confirm the suspected diagnosis of this patient's 's condition?
Rubric: The response accurately distinguishes between primary and secondary hyper- parathyroidism mentioned.
Target Capability: Differentiating between closely related sub-categories of a primary concept based on their defining features.
Target Capability Type 11:
Defining the precise rules, scope, and limitations for a given context.
Example:
Prompt: You are called to evaluate a newborn who was born yesterday 39-year-old mother. Upon examination, what chromosomal abnormality is most likely responsible for the observations typically associated with Down syndrome?
The response recommends or references karyotype analysis or equivalent genetic testing as the definitive diagnostic method for confirming the chromosomal abnormality.
Rubric:
25

---

Published as a conference paper at ICLR 2026
Target Capability:
Specifying the definitive method or standard procedure required for confirmation or validation.
Target Capability Type 12:
Communicating with clarity, adapting to audiences, and qualifying statements to reflect uncertainty.
Example:
Prompt: Which complication during pregnancy is least likely to increase the risk of postpartum uterine atonicity and why?
Rubric:
The response uses cautious and appropriate phrasing (e.g., 'least likely,' 'low association') rather than making absolute claims of zero risk.
Target Capability: Calibrating language precisely to reflect nuances and uncertainty, avoiding absolute or overly definitive statements.
Target Capability Type 13:
Applying safety disclaimers, deferring to experts, and adhering to operational constraints.
Example:
Prompt: Considering the patient's history and current presentation of sudden right arm weakness, numbness, facial drooping, and slurred speech, what is the strongest predisposing factor contributing to his condition?
The response avoids providing direct medical advice and, if appropriate, includes a disclaimer to seek immediate professional medical attention for stroke symptoms.
Rubric:
Target Capability: Adhering to predefined safety protocols or operational constraints by including appropriate disclaimers.
Target Capability Type 14:
Evaluating data significance through contextual and quantitative analysis.
Example:
Prompt: A 6-year-old boy presents with headache, cough, runny nose, and low-grade fever after being treated for a urinary tract infection with trimethorinn-ulfaanttooaaa He has leukocyte count of 2,700/mm3 with differential predominantly showing lymphocytes. What is the most likely underlying cause of his current symptoms?
Rubric: The response correctly interprets leukocyte count of 2,700/mm3 as leukopenia for a 6-year-old child (normal range 5,000/mm3 15,0000mm3).
Target Capability: Accurately interpreting quantitative data point by comparing it against a reference range to determine its significance.
Target Capability Type 15:
Systematic information filtering to maintain relevance and focus.
Example:
Prompt: A labourer involved with repair work sewers presents with fever, jaundice, and renal failure. What is the most appropriate test to diagnose the suspected infection in this patient?
The response is concise and focused on the diagnostic aspect, without excessive unrelated clinical management details.
Rubric:
Target Capability: Adhering strictly to the defined scope of a problem by excluding extrancous or irrelevant information.
Target Capability Type 16:
Systematic procedures and methodologies for formulating, testing, and validating hypothe- ses through investigation.
26

---

Published as a conference paper at ICLR 2026
Example:
Given an X-ray of a young man that shows heterotopic calcification around bilateral knee joints, what would be the next investigation help diagnose the underlying condition?
Prompt:
Rubric:
The response recommends creatine kinase (CK) testing if myositis muscle involvement is considered in the differential.
Target Capability: Proposing specific, targeted investigative actions to differentiate between hypotheses or gather further evidence.
Target Capability Type 17: Retrieving specific facts.
Example:
Prompt: Which nerves are associated with difficulty swallowing despite normal muscula- ture function, and should be tested for their functionality?
Rubric: The response identifies the Facial nerve (Cranial Nerve VII) as relevant the oral phase of swallowing (e.g., facial muscles, buccinator, taste, or saliva production).
Target Capability: Selecting specific entities from its internal knowledge that directly satisfy a given set of complex conditions.
Target Capability Type 18:
Situating information and events in historical and temporal context.
Example:
Prompt: A 10-year-old patient presents with tingling and numbness in the ulnar side the finger. Four years ago, the patient sustained elbow injury. Based on the symptoms and history, identify the fracture site that most likely occurred at the time of the initial accident.
Rubric: The response explicitly states or clearly implies that the patient's symptoms are delayed complication (i.c., tardy onset) following the initial elbow injury. Target Capability: Identifying and explicitly stating the temporal relationship between past event and a current observation.
Target Capability Type 19:
Adherence to explicit constraints on output structure, format, and content.
Example:
Prompt: 29-year-old pregnant woman at 10 weeks' gestation is experiencing progres- sively worsening nausea and vomiting, leading to significant weight loss and affecting her ability to work. Despite taking ginger and vitamin B6, her symptoms persist. Her blood gas analysis indicates pH of 7.43, pCO2 of 54 mmHg, and HCO3- of 31 mEq/L. What pharmacological intervention should added to her treatment regimen to alleviate her symptoms?
Rubric: The response provides a clear, direct, and unambiguous recommendation for the next pharmacological agent to add.
Target Capability: Formulating direct and unambiguous conclusion recommendation that resolves the primary question.
Target Capability Type 20:
Justifying conclusions by systematically eliminating alternative possibilities based evidence.
Example: Prompt:
What is the best intervention for hearing rehabilitation in a patient who has undergone surgery for bilateral acoustic neuroma?
Rubric:
The response states that conventional hearing aids and CROS/BiCROS devices
27

---

Published as a conference paper at ICLR 2026
not effective for profound bilateral sensorineural hearing loss due to bilateral cochlear nerve loss.
Target Capability: Invalidating alternative solutions by providing causal explanation for their ineffectiveness under given constraints.
K DISTRIBUTION OF MODEL SOURCES IN REFINEMENT THROUGH DIFFERENTIATION
In the refinement with four diverse and great pairs, gather responses from of SOTA models and adaptively select the best pair for RTD. We present the distribution of model sources for the responses used in the final refinement round Figure 6 The responses are drawn from diverse collection of models, with even the top model, Gemini-2.5-Pra contributing only 13.64% of the responses
pemh Mamini--..5ppp
10.39%
matratmataarmddmiinee
Mesppsshhrc
genh /ge.iii-...-ttttr 9.26%
Trewn 96, http:/mmm--00000-------eemm
opena (p26-41 7.01%%
anctrouick aude 10:.00211222200000
6.16%
anttroplck aude 20w.ett--20000022211200
6.12%
opena lost 2026-25-13 5.95%
Frewn Ric [[teppneeerr
5.62%
Other modes 6.67%
Figure 6: The Distribution of Model Sources The Final Refinement Round
28


Reviews:
[{"review_id": "jFDzTN7tpS", "reviewer": "Reviewer_aFkb", "summary": "This paper investigates the problem of reward over optimization in large language model post training and identifies that the critical factor is not general reward model accuracy but its inability to reliably distinguish excellent responses from merely good ones in the high reward tail. To address this, the authors propose a rubric based reward framework that replaces traditional Bradley Terry models with structured, interpretable rubrics consisting of weighted binary criteria evaluated by another LLM. They introduce an iterative \"Refinement through Differentiation\" (RTD) method that progressively refines rubrics by comparing high quality and diverse off policy responses generated by stronger models. Experiments across general and healthcare domains demonstrate that focusing on these high reward regions significantly improves alignment performance boosting win rates, enhancing reward accuracy in the top quality range, and delaying reward over optimization thus validating rubric based reward modeling as a robust and explainable approach to mitigating misspecification in LLM post training.", "questions": "1.  Can the rubric-based rewards that have been constructed be effective in different fields (such as general, medical, and reasoning tasks)?\n2. If the generation and verification of rubrics rely on closed-source models such as GPT-4.1, then how can we detect and correct any systematic errors in the evaluation model? \n3. For Figure 3, the green line 4 Great Pairs appears to have a better overall trend compared to the orange line 4 Great & Diverse. It is not that 4 Great & Diverse is superior to 4 Great Pairs. Please explain.\n4. Why only use one model (GPT-4.1) as the proposer instead of using different models to provide more comprehensive evidence?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Clearly identifies that the core of reward over optimization lies in the ranking accuracy at the high reward tail  rather than overall error, and provides formal theoretical support.\n- Proposes a complete pipeline combining rubric-based reward modeling, high-quality off-policy samples, and RTD iterative refinement, operationalizing “fine-grained differentiation in the high core region” into an automated and executable method.\n- From theoretical proof of the high tail’s importance, to using stronger and more diverse responses for rubric optimization, to empirical results showing that improved high tail accuracy leads to more stable RL performance with less over-optimization.", "weaknesses": "- The generation and evaluation of rubrics heavily depend on stronger closed models such as GPT-4.1, essentially a “strong teacher supervising a weaker student.” It lacks systematic validation from human experts or multi-source reviews, which risks the propagation of biases from the teacher model, especially in sensitive domains such as healthcare.\n- Heavy reliance on GPT-4.1 as the sole judge and labeler introduces an additional fragility: if GPT-4.1 systematically prefers the wrong answers that are the better response in certain patterns, these errors are directly baked into both the rubrics and the evaluation pipeline, and the method offers no robust mechanism to detect or correct such failure modes.\n- The RTD + multi-round refinement + large-scale off-policy sampling pipeline incurs substantial computational and implementation cost; moreover, the trained model may learn to mimic rubric patterns or templates instead of genuinely improving reasoning ability. The paper lacks a systematic analysis of this potential “rubric-induced reward hacking.”\n- Experiments are confined to a single model family (Qwen3-8B), a single judger GPT-4.1, and a few datasets, without comprehensive cross-scale, cross-lingual, or cross-domain validation. Furthermore, the paper does not include reasoning-intensive tasks (e.g., multi-step mathematical reasoning, code generation, or tool use), leaving it unclear whether high tail rubrics are equally effective at improving reasoning performance and mitigating over-optimization in such settings, making the conclusions more of a “strong case study” than a fully general solution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cE5BXZd0Em", "reviewer": "Reviewer_W7oY", "summary": "This paper addresses the critical issue of reward over-optimization in LLM post-training by proposing a rubric-based reward modeling approach. The authors first provide a theoretical analysis demonstrating that the accuracy of the reward model in the high-reward region is the key determinant of alignment performance. To tackle the sample inefficiency of obtaining such examples, they introduce an iterative workflow that refines scoring rubrics using off-policy responses from strong models. Experiments show that their method can mitigate reward over-optimization and improve model performance across diverse domains.", "questions": "1. Have you evaluated the generalization of your per-prompt rubrics to unseen prompts?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a rigorous theoretical analysis that clearly links reward misspecification in the high-reward tail to the phenomenon of reward over-optimization. \n2. The authors proposed a novel and practical iterative rubric refinement strategy, which effectively leverages off-policy data to create more robust reward signals.\n3. The experiments are solid. They have conducted experiments across diverse tasks and demonstrated the effectiveness of the proposed approach. The results show clear improvements in win-rate.", "weaknesses": "1. The use of a simple weighted average for aggregating rubric scores may limit the expressiveness of the reward function, although the paper acknowledges that more sophisticated aggregation schemes are an area for future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CrhK8d0z4S", "reviewer": "Reviewer_Zt7D", "summary": "This paper proposes an iterative rubric refinement framework (“Refinement-through-Differentiation”, RTD) to construct reward models that focus on the high-reward tail of LLM responses, aiming to mitigate reward over-optimization in reinforcement fine-tuning (RFT). The authors provide a theoretical justification that reward model accuracy in the top quantile of responses is the dominant factor for alignment success, and empirically show that rubric-based rewards—refined using high-quality off-policy responses—outperform baselines in both general and medical domains.\n\nWhile the motivation is compelling and the empirical results appear promising, I have significant concerns regarding bias propagation, response diversity vs. superficial style differences, and the robustness of the “pair” construction strategy. These issues undermine confidence in whether the observed gains truly reflect improved capability discrimination rather than alignment to a specific LLM’s stylistic or judgmental preferences.", "questions": "see weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Theoretical grounding**: The analysis in Section 3 provides a clean and insightful characterization of why high-reward region fidelity matters—this is a valuable contribution to the reward modeling literature.\n2. **Practical relevance**: The rubric-based approach offers a data-efficient, interpretable alternative to Bradley-Terry models, especially in domains where human annotation is scarce (e.g., healthcare).\n3. **Empirical rigor**: The ablation studies (Table 1, Figure 4) systematically validate the two proposed principles and show consistent gains across metrics.", "weaknesses": "1. The entire pipeline—initial rubric generation, top-2 selection, response scoring, and final evaluation—relies on GPT-4.1 as both proposer and judge. If GPT-4.1 systematically favors certain response styles or reasoning patterns, the refined rubric will encode and amplify this bias, leading to a self-consistent but potentially misleading reward signal.  \n\n\n\n2. The paper defines “1 Good Pair” as two responses from Gemini-2.5-Flash-Lite, and “1 Great Pair” as two from Gemini-2.5-Pro. However, sampling twice from the same model (even with temperature > 0) often yields highly similar outputs, especially on factual or constrained tasks. In such cases, the differences may be trivial (e.g., word order, minor elaboration), leading the proposer LLM to “chase noise” rather than extract meaningful quality distinctions.  \n\n\n\n3. When selecting top-2 from 16 diverse models (e.g., GPT-4o vs. Gemini-2.5-Pro), the responses often differ dramatically in length, tone, use of bullet points, emoji, hedging language, etc. While the rubric design guidelines discourage style-based criteria, the proposer LLM (GPT-4.1) may still interpret stylistic differences as quality signals (e.g., “more detailed = better”). The medical case study (Appendix I) is convincing, but it’s a single example, i hope the authors can provide broader quantitative evidence that style differences are not driving top-2 selections.\n\n\n4. It remains unclear which models dominate the top-2 selections in the final iterative rounds. For instance, does GPT-5 or o1 consistently appear in the top pair? If one model (e.g., GPT-4.1 itself) dominates, the rubric may effectively become a proxy for that model’s behavior. Could the authors provide statistics on the distribution of models represented in the final top-2 pairs across all evaluated prompts?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3juFysNhjs", "reviewer": "Reviewer_ULR5", "summary": "This paper addresses the problem of reward over-optimization in the reinforcement fine-tuning (RFT) of Large Language Models (LLMs). This issue occurs when a policy model learns to achieve high scores from a proxy reward model without actually producing high-quality output, essentially \"hacking\" the reward. To address the issue, the authors propose a rubric-based reward refinement that distinguishes excellent responses from good ones, and distinguishes among a diverse set of off-policy responses.", "questions": "1. I'm curious if there are ways to attribute the language model performance gain to specific refinement types. For example, certain refinements, such as \"Defining explicit scope, boundaries, or constraints\" mentioned in Table 3 might be particularly useful and could be explicitly incorporated into the rubric-writing instructions.\n\n\n2. Writing rubrics and generating evaluations can also be regarded as actions and optimized through reinforcement learning. With a rubric-writing LLM-as-judge, we could train it jointly with the base LLM. This could create an interesting multi-agent learning paradigm to co-evolve the rubric-writer and underlying LLMs. Any thoughts on such extensions?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is well-motivated to address the reward hacking issue by designing a rubric-based reward system. The experiments are conducted across different settings (1/4 pairs, good v.s. great v.s. great and diverse pairs). There are also analyses on refinement types to give more detailed examination of the proposed method.", "weaknesses": "The technical components—LLMs as proposers/verifiers and RFT—are not very novel \n\nThe paper's presentation could be improved. The initial motivation in Section 3 is well-reasoned but contains excessive detail better suited for the appendix. In contrast, Section 5.3 would benefit from more in-depth analysis of refinement types and their specific details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)