Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
Authors: Anisha Gunjal
Anthony Wang
Elaine Lau
Vaskar Nath
Yunzhong He
Bing Liu
Sean M. Hendryx
Venue: ICLR 2026 Poster (2026)
Keywords: representation learning for language
datasets and benchmarks
reward modeling
reinforcement learning
natural langauge processing
large language models
reasoning
alignment
Primary area: foundation or frontier models, including LLMs
TL;DR: An on-policy RL framework that uses rubric-guided rewards for training LLMs on real-world reasoning tasks.

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for complex reasoning tasks with clear correctness signals such as math and coding. However, extending it to real-world reasoning tasks is challenging, as evaluation depends on nuanced, multi-criteria judgments rather than binary correctness. Instance-specific rubrics have recently been used in evaluation benchmarks to capture such judgments, but their potential as reward signals for on-policy post-training remains underexplored. We introduce $\textbf{Rubrics as Rewards (\textit{RaR})}$, an on-policy reinforcement learning method that extends RLVR beyond verifiable domains by using rubric-based feedback. Across both medical and science domains, we evaluate multiple strategies for aggregating rubric feedback into rewards. The best RaR variant achieves relative improvements of up to 31\% on HealthBench and 7\% on GPQA-Diamond over popular LLM-as-judge baselines that rely on direct Likert-based rewards. These results demonstrate that RaR-trained policies adapt well to diverse evaluation formats, performing strongly on both rubric-based and multiple-choice tasks. Moreover, we find that using rubrics as structured reward signals yields better alignment for smaller judges and reduces performance variance across judge scales.

Published as a conference paper at ICLR 2026
RUBRICS AS REWARDS: REINFORCEMENT LEARNING BEYOND VERIFIABLE DOMAINS
Anisha Gunjal
Anthony Wang" Elaine Lau"
Vaskar Nath" Yunzhong He
Bing Liu Sean Hendryx*
Scale AI
anisha. wwwsssagnnjalssale..co
ABSTRACT
Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for complex reasoning tasks with clear correctness signals such as math and cod- ing. However, extending it to real-world reasoning tasks is challenging, uation depends on nuanced, multi-criteria judgments rather than binary correct- ness. Instance-specific rubrics have recently been used in evaluation benchmarks to capture such judgments, but their potential as reward signals for on-policy post- training remains underexplored. We introduce Rubries as Rewards (RaR), an on-policy reinforcement learning method that extends RLVR beyond verifiable do- mains by using rubric-based feedback. Across both medical and science domains, we evaluate multiple strategies for aggregating rubric feedback into rewards. The best RaR variant achieves relative improvements up to 31% % on HealthBench and 7% on GPQA-Diamone, over popular LLM-aa-juudee baselines that rely on direct Likert-based rewards. These results demonstrate that SaR-trained policies adapt well to diverse evaluation formats, performing strongly on both rubric-based and multiple-choice tasks. Moreover, we find that using rubrics as structured re- ward signals yields better alignment for smaller judges and reduces performance variance across judge scales.
eval-
INTRODUCTION
Reinforcement Learning with Verifiable Rewards (RLVR) has enabled large language models to elicit complex reasoning tasks with clear verifiable outcomes. This is especially effective in domains like math and code, where reward models can be replaced by scoring functions or test cases that automatically verify correctness (Lambert et al. 2024 Guo et al. 2025a Cui et al. 2025) However, extending RLVR to unstructured, real-world reasoning is challenging because such tasks lack easily verifiable answers. A common workaround is to preference- based reward models, but they tend to overfit superficial artifacts (e.g. response length, formatting quirks, annotator biases)
(Singhal et al. 2023 Wang al., 2024a; Chen et al., 2024bb Ye al.
2024 Gudibande et al.)
2023) and require large volumes of pairwise comparisons (Ouyang et al.
2022). Instance-specific
rubrics have recently emerged for nuanced evaluation in expert domains (Arora al. 2025). yet
their application in on-policy training for expert-level reasoning is largely unexplored.
To address this gap, we explore a paradigm shift that introduces a middle ground between the sim- plicity of verifiable rewards and the expressiveness of preference rankings, which often come with human artifacts and operational overhead. We introduce Rubrics as Rewards (RaR), a frame- work for on-policy Reinforcement Learning that uses structured criteria or rubrics as the core re- ward mechanism. Rather than using rubrics only for evaluation (Arora et al.) [025), (Sirdeshmukh) al. 2025), we treat them as checklist-stlle supervision that produces reward signals for on on-policy RL. Each rubric is composed of modular, interpretable subgoals that provide automatable feedback aligned with expert intent. By decomposing "what makes good response" into tangible, human- interpretable criteria, rubrics offer middle ground between binary correctness signals and coarse preference rankings.
Work conducted while Scale AI.

---

Published as a conference paper at ICLR 2026
Rubric Generation
GRPO with Rubric as Rewards
Etunan Experts
Prompt
pehent perents
Policy Model
Reloea
lilly dingnosic?
Advetage
Computaion
-diagnosis>
Reward
Fogur Greending
Impprrunce
Imgortant: <Jey sign-
Important: "guntitaiiv finding:
SelF-contirnd
Uoorngg
Rubrics
Mention...
L Judge
Prifal:: nisdiagnosie
Figure I: Overview of Rubrics as Rewards (RaR). (i) Rubric Generation: We synthesize prompt- specific, self-contained rubric criteria using strong LLM guided by four core design principles, with reference answers serving as proxies for expert supervision. (ii) GRPO Training: These rubrics are used prompt an LLM judge for reward estimation, which drives policy optimization via the GRPO on-policy learning loop.
Previous works train gencrative reward models that learn to evaluate reasoning or final outputs with interpretable scores (Chen et al., 2025 Whitehouse al. 2025 Anugraha al., 2025 Guo et al. 2025b), and some have even used model's internal confidence estimates as a proxy for reward (Zhao al. 2025). More recent efforts have extended verifiable datasets beyond STEM domains, broadening the applicability of RLVR methods to a wider range of tasks (Su et al.) [0025]] Ma al. 2025). Yet general-purpose approach for specifying reliable reward signals remains elusive, particularly in tasks without a single ground truth where both subjective and objective criteria must be considered. In contrast, we rubrics as instance-specific, reusable reward functions. Once generated, rubrics provide interpretable and automatable supervision that can be applied consistently across new rollouts, offering scalable and transparent alternative to opaque reward modeling in on- policy learning.
Recent concurrent works explore checklists and principled rubric criteria for preference tuning and LLM safety (Gallego 2025 Viswanathan et al. 2025 Dineen et al., (025), highlighting growing trend toward structured supervision. contrast, we convert rubrics into reward functions for on- policy RL, targeting expert reasoning and applied real-world domains. This closes the rubric-to- learning loop and improves performance on both cubric-guided evaluations and tasks with verifiable answers. Figure iillustrates our framework.
Our key contributions are as follows: (i) We introduce Rubrics as Rewards (RaR), an on-policy reinforcement learning framework that uses shecklist-style rubrics for multi-criteria supervision in reasoning and real-world domains. (ii) We synthesize instance-specific rubrics for medicine and science and release the corresponding training sets, ReR-MMeiiiine and RaR-Science. (iii) RaR- trained models consistently outperform strong baselines and yield a stable, generalizable training signal, with gains both rubric-scored and verifiable multiple-choice evaluation settings. (iv) Our results demonstrate that rubric-based rewards provide stable supervision across judge sizes, helping smaller models align effectively with human preferences and maintaining robust evaluation performance from small to large judges.
RUBRICS AS REWARDS
2.1 PROBLEM FORMULATION
Let denote an input prompt and ÿ TT(- |x) be a sampled response from a model parameterized by 0. In domains without single ground-truth answers or automatic correctness signals, we define structured reward function using instance -estacc--ppecffc rubric criteria.
Datasets released http:////hhaaeeeeeeooooottaaaaaa
2

---

Published as a conference paper at ICLR 2026
Each prompt is associated with set of lo rubric items (ww,,;))) where wy R denotes the weight of criterion j. and Cj (x, )) + {0, 1) is binary correctness function that indicates whether the response y satisfies that criterion given the prompt.
2.2 REWARD AGGREGATION STRATEGIES
We investigate complementary approaches for combining rubric feedback into scalar rewards:
Explicit Aggregation. Each criterion is independently evaluated using an LLM-as-judge, and the final normalized reward is computed as:
r(x, () Σn= wy . (x,
Σラ=1 w/
Normalization makes rewards comparable across prompts that differ in rubric count or weights. Although we use binary checks for ej in our experiments, the formulation can be extended to continuous-values scores.
Implicit Aggregation. All rubric criteria along with categorical weights passed to an LLM- as-judge, delegating the aggregation to the model itself to produce a single scalar reward:
(2) Here, fo denotes an LLM--based judge that takes the prompt æ, the response d. and the set of rubric criteria {d,} input. This formulation allows the model to compute holistic reward score directly, avoiding the need to manually tune rubric weights.
Timplicin xz, 9) = fo (z, d. {dz))=z)
The prompts used for each method are detailed in Appendix A.7
2.3 GENERALIZATION OF RLVR WITH RUBRICS AS REWARDS
Rubric-based reinforcement learning extends the standard RLVR (Reinforcement Learning with Ver- ifiable Rewards) setting by supporting multi-dimensional, prompt-specific evaluation criteria. We formalize this relationship below.
Remark (Rubrics as Rewards subsumes RLVR). The RLVR setting is special case of rubric- based rewards defined in Equation | where verifiable correctness function that compares the model output ý against the known correct answer For example, this could involve exact match or test case execution. Formally:
1, w1 = 1, and Ci (æ, () reduces to single
TRLVR (I. 9) = match(y. ù)
(3)
where match(y, 9) E {0, }} indicates whether the response satisfies the verifiable correctness con- dition.
Rubric-based reward functions thus generalize RLVR by enabling multi-diimensoonll supervision, flexible weighting across criteria, and the incorporation of both objective and subjective aspects of response quality. This formalization highlights that RLVR can be seen as a restricted instance of rubric-guided RL with single essential criterion. In contrast, rubric-based rewards further enable structured supervision in settings where correctness is multifaceted and may not be strictly verifiable.
3
RUBRIC GENERATION
3.1 DESIDERATA
A rubric specifies criteria for high-quality responses and provides human- humaa--nterppeetable supervision. We identify four desiderata for effective rubric generation:
Grounded in Expert Guidance. Rubries should reflect domain expertise by capturing the essen- tial facts, reasoning steps, and conclusions necessary for correctness. Ideally, this grounding comes from human experts their high-quality proxies.
3

---

Published as a conference paper at ICLR 2026
Comprehensive Coverage. Rubrics should span multiple dimensions of response quality, includ- ing factual accuracy, logical coherence, completeness, style, and safety. Negative criteria (pirfalls) help identify frequent high-risk enrors that undermine overall quality.
Criterion Importance. Rubrics should reflect that some dimensions of response quality are more eritical than others. For example, factual correctness must outweigh secondary aspects such as stylistic clarity. Assigning weights to criteria ensures this prioritization, whether through simple categorical tags, explicit numeric values, or learned weighting schemes.
Self-Contained Evaluation. Each rubric item should be independently actionable, allowing either human annotators automated judges assess it in isolation without requiring external context domain-specific knowledge.
3.2 RUBRICS CREATION
We apply these desiderata datasets for reasoning tasks in medicine and science. Given the scarcity of human-annotatee rubric datasets in these domains, we use LLMs to generate instance-specific rubrics from golden reference answers scale, enabling the study of structured rewards without costly human annotation.
For each prompt, an LLM generates rubric of 7-20 self-contained items. Each item is assigned both a numeric and a categorical weight reflecting its relative importance. While numeric weights provide fine-grained prioritization,
our experiments we adopt categorical labels (Exsential, Important, Optional, Pitfall) for ease of implementation and interpretability in controlled settings. The resulting rubrics are then used directly as reward functions through either explicit aggregation (Eq. I or implicit aggregation (Sec. 2.2
In practice, we generate rubrics using OpenAl's 03-mini and GPT-4o OpenAl o3-mini 2025 Jaech et al., 2024| Hurst et al. 2024). conditioning generation reference answers from the underly- ing datasets to approximate expert grounding, The resulting collections- Rictiions-RRR--ae RaR-Medicine and RaR- Science- released publicly. These rubric sets supervise policy models with GRPO using both explicit and implicit reward aggregation.
EXPERIMENTS
4.1 DATASETS
We investigate the utility of rubrics as rewards across two reasoning domains, medicine and science.
RaR-Medicine: A dataset of 20k prompts drawn from diverse medical reasoning sources, includ-
ing madicall-o--resonn ng- SFT (Chen et al. 2024a), natural_reasonin (Yuan et al. 2025).
SCP-116K (Lu al. 2025). and Thonhral/houghh-4000 (General Reasoning 2025)
Instance-
specific rubrics for this dataset are generated with GPT-4o (see Appendix A.2)
RaR-Science: A dataset of ~20 prompts curated to align with GPQA-
categories.
Prompts are sourced from natural reasoning (Yuan et al., 2025). SCP- 16K (Lu et al. 2025 and Genera/Thoought-430K (General Reasoning 2025). covering a broad range scientific rea-
soning tasks (Appendix A.3). Rubrics for this dataset synthesized with 03-mini.
4.2 TRAINING DETAILS
We conduct all experiments using on-policy reinforcement learning with the GRPO algorithm (Shao et al. (2024), taking Qwen2.5-77 the base policy. Models trained with batch size of 96, learning rate of X 10 6 and constant schedule with 10% lincar warmup. Complete hyperpa- rameter settings are listed in Appendix A.4 Training runs executed single compute node equipped with NYIDIA H100 GPUs
Our training pipeline consists of the following key components:
Response Generation: For each prompt q. we sample = 16 responses from the current policy To- using a context length of 3584 and sampling temperature of 1.0.
4

---

Published as a conference paper at ICLR 2026
Reward Computation with Rubries: We use gpt-4o-mini the judge model to assign rewards Rg to the sampled responses. We experiment with various reward computation and aggregations strategies further described in Sections and
Policy Update:
The policy weights are updated using GRPO based on the computed rewards.
Category
Method/Baseline
Trained
Aggregation Type
Reward Grounding
Rubric-Frec
OFF-THE-SHEE
N/A
NA N/A
Rubric-Free
DIRECT-LIKERT
Single Likert score
Rubric-Free
REFERENCE-LIKER
Single Likert score
Reference answer Instance-agnostic rubrics Instance-specifie rubrics
Rubric-Based
RaR-PREDEF INED
V
Explicit aggregation
Rubric-
RaR-EXPLICIT
Explicit aggregation
Rubric- Based
PMRIIMPLICIT
Implicit aggregation
Instance specific rubrics
Table 1: Overview of rubric-free and rubric-basee methods and baselines.
4.3 RUBRIC-FREE BASELINES
We consider various rubric-free baselines and off-the-shelf post-trained models. Rubric-free base- lines trained with Qwen2. 5-7B the base policy.
OFF-THE-SHELF: For off-the-shelf baselines we evaluate performance on Qwen2. 5-7B. We also in- clude the performance of Qwen2. 5-7B-Inst compare with instruction-tuncl variant of the base policy.
DIRECT-LIKERT: An -u- provides a direct assessment for each response-prompl pair on 1-10 Likert scale (Zheng et al. 2023 Kim et al. 2024), normalized to [0, 1j. The resulting score is used directly as the reward signal for training.
REFERENCE-LIKERS An LLM-as-judge compares the generated response against a reference answer (written by experts stronger LLMs) and assigns a 1-10 Likert score (Zheng et al.) (002)), nor- malized [0, 1]. This reference- guided score used the reward signal for policy updates. The reward for each (prompi, response, reference) triplet is defined as:
Ra x) Norm (Lemmmilettttooregg.. x,,,""))
where x² denotes the reference answer.
4.4 RUBRIC-GUIDED METHODS
This method uses fixed set of generic rubrics for all prompts (e.g. response is concise, response contains correct information). It employs the Explicit Aggregation method (Equation]) D with all criteria weighted uniformly (see Appendix A.
PaR-PREDEFINED:
PaR-EXPLICIT This variant also uses Explicit Aggregation using weighted (Equation I but applics it to instance-specifie rubrics from Section We manually assign numerical weights based on the generated categorical labels: ("Essential": 1.0, "Important" 0.7, "Optional": 0.3, "Pitfall": [
RMR-IMPLICIT: This variant uses the Implicit Aggregation method (Equation 2). It leverages prompt- specific rubrics, where judge model evaluates the response as whole to assign a single Likert rating (1-10), avoiding the need for hand-tuned weights. The reward normalized the [0, 1] range during training.
An overview of all baselines and methods is summarized in Table
4.5 EVALUATION SETUP
Rubric-Based Evaluation We evaluate models trained with RaR-Medicine on Health- Bench (Arora et al. 2025), benchmark of 5,000 clinical conversations designed to assess model
Pitfall criteria are phrased in positive form (e.g., "The response avoids misinformation",. so satisfying them contributes positively the score. If pitfall is satisfied, the corresponding reward is reduced penalized.
5

---

Published as a conference paper at ICLR 2026
HoalthBench
GPQA Diamond
70
60
35.0
34,8
36.5
36.9
31.7
31.7
Score (8)
40
30-
30
22.7
25.5
28.9
29.7
31.2
Mean Accuracy
20
20
12:5
10
10
1.7
Oven2.5. Owen2.5-
Reference- Likert
Raf- Implicit
Owen2.5 8-70
Owenz Te-inatrcct
Divect-Lila Likert
Reference Likert
RaR Predefined
Rart Explint
Ran Rof-iimplint
7b- Instruct
Likert
Predefined
Communication quality
Accuracy
Completeness Overall score
Instruction following
Context awareness
Figure 2: Performance of baselines and RaR (Rubrics Rewards) variants for the medicine and sci- ence domains. HealthBench (left): shows per-axis scores across five core axes, with a thin dashed gray line indicating the overall score (all values shown percentages). GPQA-Diamond (right): mean accuracy over 10 runs; error bars represent 95% confidence intervals. All policies are evalu- ated using gpt-40-mini as the LLM-as-Judge. Across both domains, RaR-Implici consistently outperforms Direct-Likert and demonstrates a competitive advantage over Reference-Likert.
safety and helpfulness in realistic medical scenarios. Performance is measured using detailed, physician-authoree rubrics. We generate responses with greedy decoding (temperature 0) and report both overall scores and per-axis scores following the original setup. For ablation studies, sample subset of 1,000 prompts (hereafter referred as HealthBench-Ikk 1k) and use the rest for training.
Multiple-Choice Evaluation Each model is evaluated across 10 independent runs, using greedy decoding (temperature=0) to sample one response per prompt. Answer choices are permuted per example to reduce positional bias, and outputs are parsed for boxed answer formats (e.g., boxed(A)) extraction fails, we fall back to a GPT-4o verifier that checks whether the response contains the correct option letter or text (see Appendix A.5). Final accuracy is reported as the mean over 10 runs, and we include 95% confidence intervals to account for run-to-run variance.
LLM-Judge Alignment Evaluation To measure how well LLM judges align with human prefer- ences, we build paired evaluation set from roughly 3,000 HealthBennh prompts. For each prompt, we take the practitioner-appre tive via controlled edits (see Appendix A. for method used for perturbation and prompt selection). The metric is pairwise preference accuracy i.e. the fraction of pairs where the preferred response scores higher reported across judge models of varying sizes.
answer as the preferred response and create perturbed alterna-
5 RESULTS
We now present the main findings of our study.
Rubries as Rewards shows strong gains across evaluation settings. Table reports results on HealthBench (rubric-based, free-form) and GPQA-Diamond (multiplc-choice). RaR PRR-mmiiiit con- sistently outperforms Direct-Likert, with relative gains up to on HealthBench and 7% on GPQA. Both rubric-guided variants achieve higher scores than the base and instruction-tumed poli- cies. Gains on GPQA-Diamond show that rubric-induced skills generalize beyond rubric-based eval- uation. The RaR-Predef fned variant, which applies a fixed list of generic rubrics to every prompt (no eetaace-specifie synthesis), underperforms because generic criteria miss prompt-specific re- quirements and common failure modes, producing misaligned reward signals. Hence, effective training requires -psccfc rubric synthesis as they better capture task context and typical failure modes.

---

Published as a conference paper at ICLR 2026
LLM Judge Alignment Across Model Scales
1.0
0.8
Accuracy
0.6
0.4
Pure Likert
Rubrics
0.2
Rubrics w/o reference answer
0.0
3b-judge
7b-judge
14b-judge
32b-judge
72b=judge
gpt-4o-mini
63-mini
gpt-40
Model
Figure 3: Alignment Study of LLM Judges across Model Scales. Rubrics as Rewards (orange) consistently improves alignment with human preferences across LLM judge sizes compared to direct Likert-based scoring (blue). Judge Alignment using synthetic rubrics without expert grounding (green) outperform the direct Likert baseline, but still fall short of expert-grounded rubries (orange). The rubric structure especially benefits smaller judge models, helping them close the gap with larger models when guided by checklist-style criteria.
Beyond these gains, RaR-Implic icit also shows small but consistent gains over Reference-Likert In our setup, rubrics are generated with stronger LLMS using reference answers as proxies for expert supervision, so rubric quality impacted by reference quality. Even so, converting open-ended answers into explicit criteria yields effective, well-aligned reward signals.
Between the two rubric-guided methods, RaR-Implicit attains the strongest results overall; fixed weighted sums in BaR-Explicit offer more control but can be brittle. Explicit weighting can be difficult tune but offers greater interpretability; we view the choice as peplication-depended and leave it to practitioners. Future work could explore learned or dynamic weighting strategies that maintain interpretability while improving adaptability.
Rubrics enhance alignment with human preferences across model scales We evaluate align- ment with humans by having LLM judges of varying sizes score chosen vs. rejected HealthBench- Ik responses on 1-10 scale under two settings: (i) aubric-guided (RAR-IMPLICIT)) where the instance- specific rubric is provided, and (ii) rubric-free (DIRECT-LIKER) where only the prompt and answers are shown. Figure reports pairwise preference accuracy (the fraction of pairs where the preferred response receives the higher score). Rubric guidance improves accuracy for every judge size, with the largest gains for smaller judges, narrowing the gap to larger models. This indicates that explicit, context-specific criteria help judges distinguish subtle quality differences better than direct Likert scoring. Further analysis of judge-scale effects on GRPO training is detailed in Appendix A.9
Expert guidance is crucial for synthetic rubric generation Human guidance significantly in- fluences the effectiveness of rubrics in capturing subtle human preferences. Figure highlights performance differences between rubric-based evaluations that include reference answers and those without them. The data shows that rubrics developed with reference answers achieve higher accu- racy, emphasizing that human insights integrated during rubric generation enable granular criteria and improved alignment with human preferences.
6 ABLATIONS AND ADDITIONAL ANALYSES
Impact of Rubric Generation Strategies in Real-World Domains How does the method of rubric generation affect downstream training in challenging, real-world settings? To investigate this, we hold out HealthBench-11 1k for evaluation and use 3.5k prompts from the remaining Health- Bench pool to generate rubrics for training as it has access to human generated rubrics. Results are summarized in Table
7

---

Published as a conference paper at ICLR 2026
Training Method
Overall Score
Expert-Answer-SPi
20.4%
Simple-Likert
23.9%
Reference-Liken
31.7%
32.0%
RaR-Implicit-Synthee
35.9%
RaR-Implicit-Human
34.8%
Table Evaluation on HcalthBenchh Comparison of human- vs. synthetic-generale rubrics (with and without reference answers). RaR methods trained with GRPO significantly outperform Likert- only, Reference-based-lieeee based-likert and SFT baselines. Synthetic rubrics generated without access to ref- erence answers perform notably worse, highlighting the importance of human-gras -aomandd guidance. Notably, human-authored rubrics and synthetic rubrics with access to references yield comparable performance.
Ablation Setting
Overall Score
Essential-Only Rubrics
34.9%
No Categorical Labels
38.8%
No Pitfall Criteria
37.2%
All Rubrics
37.2%
Table 3: Ablation results for elements of rubric design on HealthBench--1 (trained on HealthBench- 3.5k subset with Qwen2.5-7B base policy). Rubrics generated using 03-mini with access to refer- ence answers.
In-domain testing on HealthBench-- amplifies RaR's gains: every instance-specific rubric-based method outperforms rubric-free baselines. Notably, even the weakest RaR variant significantly surpasses Reference-Likert. underscoring the advantage of structured supervision in subjective, open-ended domains like healthcare. We attribute this to the finer granularity and clarity rubrics pro- vide in assigning rewards-especially when correctness is not binary and answers may vary in tone, completeness, or safety relevance.
Moreover, we find that rubric quality is crucial: synthetic rubrics generated with reference-answer guidance consistently outperform those generated without it. This highlights the importance of incorporating expert signal, whether via human-in-the-loow -oop-e-oopp annotations or high-quality reference completions, for generating effective and aligned rubrics. Purely synthetic rubrics, while scalable, currently fall short in capturing the subtle criteria required for robust training in high-stakes domains.
Elements of Rubric Design This ablation study examines how the structure and weighting of synthetic rubrics affect downstream performance on HealthBench-Ik. As shown in Table rubrics that include a broader range criteria outperform those limited to essential checks, suggesting that richer evaluation signals lead to better learning. Interestingly, ferences when including rubric weights or pitfall criteria during training. One possible explanation is that synthetically generating effective pitfall criteria is inherently difficult, it requires antici- pating the most common or critical failure modes of the model, a task that often demands human intuition and domain expertise. As result, these synthetic negative criteria may lack the specificity or relevance needed to meaningfully penalize undesirable responses.
observe minimal performance dif-
To assess how the capabilities of rubric-generatin LLMs affect downstream performance, generate synthetic rubrics without access to reference answers and use them to train policies on HealthBench. This isolates the effect LLM quality on reference-free rubric utility. Specifically, evaluate the HealthBench-11 kk subset, using models trained on rubrics generated for the remaining 4k training examples from HealthBench.
Impact LLM Expertise on Rubric Quality
As shown in Table larger more capable LLMs generally produce more effective rubrics, with GPT-4o yielding the best performance among reference-fre models. However, all remain below the performance of rubrics generated with reference guidance (e.g., 03-mini with access to reference
8

---

Published as a conference paper at ICLR 2026
Rubric Generation Model
Overall Score
03-mini (Rubrics with reference)
35,9%
GPT-4o
34.2%
GPT-4o-mini
32.7%
03-mini
32.4%
Dwwn-72B-Instrren
32.7%
Qwen-32BBInntrucc
31.1%
Qwen-7B-Instruee
31.9%
Table 4: Policy performance HcalthBench-lk when trained with GRPO using rubrics generated by different LLMs without reference answers. GPT--4o-generated rubrics yield the strongest perfor- mance, though they still fall short of rubrics generated with expert inference-guidede supervision. Smaller aligned models (e.g., GPT-4o-mini, O3-mini) remain competitive with larger open-weight models, underscoring the importance of alignment and reasoning ability in rubric generation.
Baseline/Method
Overall Score
Qwen2.5-3B (Base Policy)
4.13%
Direct-Likert
13.74%
Referrenc--iken
17.95%
RaR-Implicit (ours)
21.55 %
Table 5: Performance HealthBench-Ik Ik comparing methods trained an alternative policy model (Qwen2. 5-3B) shows that rubric-based reward training (RaR-Implicit) continues to outper- form Likert-basse based baselines.
answers). Additionally, model attributes such as instruction tuning and reasoning capabilities play a key role in the effectiveness of rubric generation.
Robustness to policy model choice To assess whether our findings on rubrics as rewards remain agnostic to policy models, we we repeat the training procedure using smaller base model, Owen2.5-333 We train this model on the HcalthBench-33 prompts using Direct-Likert, Reference-Liker and RaR-Implicit t, and evaluate the held-out HealthBench-lk set. As shown in Table the relative performance trends are consistent with the 7B model: DRR-Imlicci achieves the strongest performance among the compared methods. This provides additional data point suggesting that rubric-based rewards remain effective when applied to a smaller policy models.
RELATED WORK
RLVR across domains Reinforcement learning with verifiable rewards (RLVR) is expanding be- yond math and code. GANNEALL-REASOEE trains 200k mixed corpus spanning physics, finance, and policy, and reports ten-point gain on MMLU-Pro after GRPO fine-tuning (Ma et al.) 2025). A follow-up extends RLVR to medicine, chemistry, psychology, and economics, showing that a single cross-domain reward model can supervise all four without task-specific tweaks (Su al. 2025a). In healthcare, MED-RLVE applies similar methods to multiple-choice clinical QA, improving accuracy over supervised baselines while eliciting chain-of-thon
from a 3B base model (Zhang et 2025) These results indicate steady progress, yet sparse signals, verifier reliability, and limited benchmark coverage remain open challenges.
Rubrics for evaluation and training Task Task-specific rubrics are increasingly used evaluate LLMs in difficult-to-verijs domains (Arora et al.) 2025: Ruan et al. 2025] Hashemi et al.) [024] Pathak jet al. 2025). Pathak et al. show that rubric-prompted LLM graders are more accurate and con- sistent than question-agnostic checklist (Pathak et al.) 2025) HEALTHBENCE
scales this idea in medicine, pairiiin 48k clinician written criteria with GPT-4 judges to score various axes (Arora et al, 2025). Beyond evaluation, rubrics used to condition preference pairs for DPO (CPT; (Gallego, 2025) ) and to guide checklist-basee preference tuning in safety, instruction instruction-follovin. following. and creative-
writing settings /Viswanathan al. 2025 Dineen al. 2025
Kim al. 2025 These lines of

---

Published as a conference paper at ICLR 2026
work primarily use rubrics to grade outputs or to condition preference data, often in non-reasoning domains such as safety, instruction following, or creative writing. In contrast, we use rubric criteria directly reward signals for on-policy RL in expert-reaoonngg and real-world domains.
Learning from feedback signals RLHF trains policies with large numbers of human compar- isons, which introduces subjectivity and can lead to reward hacking (Ouyang al., (022). RLVR reduces these issues by using programmatic checks, from exact matches on GSM8K and MATH to mixed-domain verifiers in GENERAL- REAERALRRESOOER and CROSS-DOMANE RLVR (Ma al. 2025 Su et al., 2025a), although signals can be sparse. Process supervision (ighhman et al. 2023) pro- vides denser guidance via step-level labels, and MCTS ACCTS-generated annotations or generative reward models such as THINKPRM improve performance, but with high annotation cost (Li et al. 2025
Khalifa et al. 2025). Rubrie-based RL finds middle ground by turning multiple rubric criteria into structured verifiers and using their scalar scores as denser rewards.
8 CONCLUSION
We introduced Rubrics as Rewards (RaR), framework for post-training language models us- ing structured, checklist-style rubrics as reward signals. By decomposing response evaluation into transparent, multi-criteria objectives- subjective and objective- RaR provides a modular and interpretable alternative to preference-based methods. Our experiments demonstrate that rubric- guided training achieves strong performance across domains, significantly outperforming Likert- based baselines and matching exceeding the performance of reference- based reward generation. Additionally, we show that RaR improves alignment with human preferences while reducing depen- dence large judge models.
LIMITATIONS AND FUTURE WORK
Our work focuses on medicine and science to enable controlled experiments. This choice allows us controlled experiments, but broader validation across dialogue, tool use, other agentic tasks remains an important direction. We evaluate only two reward aggregation strategies, implicit and explicit, since they capture complementary extremes of flexibility and control; future work could explore more advanced ways of combining rubric criteria, such as learning continuous weights for each criterion dynamically adjusting weights over the course of training mimic curriculum (e.g., prioritizing essential correctness early, then gradually emphasizing more subtle qualities like style or safety). Finally, we use off- tffflf LLMs as judges for accessibility and reproducibility; exploring specialized evaluators with stronger reasoning generative reward models may yield further gains.
REPRODUCIBILITY STATEMENT
Our datasets RaR-Medicine and RaR-Science datasets with per-prompt rubrics are released at https://hugging:
We document data construction and
rubric synthesis in Sections 2 and with full prompt templates in Appendices A.6 A.8 and 10
Training settings for on on-policy GRPO (model, batch size, schedule, hardware) appear in Section 4.2 and Table 12, All judge prompts used to compute rewards for rubric-guided and Likert baselines
in Appendix A.7 Evaluation protocols, metrics, and prompts are detailed in Section 4.5 and Ap- pendix A.5] report means with 95% confidence intervals and use the open-source simple-eval. framework for evaluation.
ACKNOWLEDGMENTS
We thank Qin Lyu, Nikhil Barhate, and Zijian Hu for supporting this research through the devel- opment of the in-house bost-training platform RLXE. We also thank Robert Vacareanu for valuable early feedback and discussions.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
David Anugraha, Zilu Tang, Lester James V Miranda, Hanyang Zhao, Mohammad Rifqi Farhan- syah, Garry Kuwanto, Derry Wijaya, and Genta Indra Winata. R3: Robust rubric-agnostic reward models. arXiv preprint arXiv:2505.13386 2025.
Rahul K Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero- Candolla, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, et al. Health- bench: Evaluating large language models towards improved human health. arXiv preprint DrXiv:2505.08775. 2025.
Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang. Jianye Hou, and Benyou Wang. Huatuogpt-ol, towards medical complex reasoning with llms. arXiv preprint arXiv:2412.18925, 2024a.
Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf. arXiv preprint orXiv:2402.07319, 2024b.
Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang. Tong Zhang, et al. Rm-rl: Reward modeling as reasoning. arXiv preprint 250.0:2500.02387 2025.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang. Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456 2025.
Jacob Dineen, Aswin RRV, Qin Liu, Zhikun Xu, Xiao Ye, Ming Shen, Zhaonan Li, Shijie Lu, Chitta Baral, Muhao Chen, al. Qa-lign: Aligning Ilms through constitutionally decomposed qa. arXiv preprint wrXiv:2506.08123, 2025.
Víctor Gallego. Configurable preference tuning with rubric-guided synthetic data. arXiv preprint arXiv:2506. 17702, 2025.
General Reasoning. General reasoning. 2025. URL httpp:://g..innn inc/
Arnav Gudibande, Eric Wallace, Charlic Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary llms. arXiv preprint 1.Xiv:2305.15717.0 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint 129499,,001.,248, 2025a.
Jiaxin Guo, Zewen Chi, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, and Furu Wei. Reward reasoning model. arXiv preprint 2rXiv:2505.14000 2025b.
Helia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme, and Chris Kedzie. Lim-rubric: A multidimensional, calibrated approach to automated evaluation of natural language texts. arXiv preprint www.xx2250.00277. 2024.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-40 system card. arXiv preprint arXiv:2410.21227, 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameyy, et al. Openai ol system card. arXiv preprint arXiv:2412.1677 2024.
Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jackyeom Kim, Hao Peng, Moon- tae Lee, Honglak Lee, and Lu Wang. Process reward models that think. arXiv preprint ar5iv:2504. 16828, 2025.
11

---

Published as a conference paper at ICLR 2026
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.01533 2024.
Zac Myung Kim, Chanwoo Park, Vipul Raheja, Suin Kim, and Dongyeop Kang. Toward evaluative thinking: Meta policy optimization with evolving reward models. arXiv preprint arXiv:2504.20155 2025.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T," ulu 3: Pushing frontiers in open language model post-training. arXiv preprint 1xXi:2441.15//22 2024.
Shuangtao Li, Shuaihao Dong, Kexin Luan, Xinhan Di, and Chaofan Ding. Enhancing reasoning through process supervision with monte carlo tree search. arXiv preprint arXiv:2501.0144 2025.
Hunter Lightman, Vincet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Dakuan Lu, Xiaoyu Tan, Rui Xu, Tianchu Yao, Chao Qu, Wei Chu, Yinghui Xu, and Yuan Qi. Scp-116k: A high-quality problem-solee solution dataset and a generalized pipeline for automated ex- traction in the higher education science domain. arXiv preprint arXiv:2501. 2025.
Xueguang Ma. Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. Gencral-reasoner: Advancing Ilm reasoning across all domains. arXiv preprint arXiv:2505. 2025.
OpenAl 03-mini. Openai o3-min, 2025. URL http:p////ooeeeaa...cc//////a/ap
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Aditya Pathak, Rachit Gandhi, Vaibhav Uttam, Yashwanth Nakka, Aaryan Raj Jindal, Pratyush Ghosh, Arnav Ramamoorthy, Shreyash Verma, Aditya Mittal, Aashna Ased, et al. Rubric is all you need: Enhancing Ilm-based code evaluation with question-specific rubrics. arXiv preprint arXiv:2503.2399. 2025.
Jie Ruan, Inderjeet Nair, Shuyang Cao, Amy Liu, Sheza Munir, Micah Pollens-Demper Tiffany Chiang, Lucy Kates, Nicholas David, Sihan Chen, et al. Expertlongbench: Benchmarking language models expert-level long-form generation tasks with structured checklists. arXiv preprint arXiv:2506.0124) 2025.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023.
Ved Sirdeshmukh, Kaustubh Deshpande, Johannes Mols, Lifeng Jin, Ed-Yeremai Cardona, Dean Lee, Jeremy Kritz, Willow Primack, Summer Yue, and Chen Xing. Multichallenge: A realis- tic multi-turn conversation evaluation benchmark challenging to frontier llms. arXiv preprint arXiv:2501.17399 2025.
Yi Su, Dian Yu, Linfeng Song. Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.2388 2025a.
Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu. Min Zhang, and Dong Yu. Expanding with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.23893 2025b.
12

---

Published as a conference paper at ICLR 2026
Vijay Viswanathan, Yanchao Sun, Shuang Ma, Xiang Kong, Meng Cao, Graham Neubig, and Tong- shuang Wu. Checklists are better than reward models for aligning language models. arXiv preprint arXfv:2507.18624, 2025.
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of Ilms for diverse user preferences: Directional preference alignment with multi-objective rewards. arXiv preprint arXiv:2402.18577 2024a.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Aruiraj, Xuan He, Ziyan Jiang, al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances Neural Information Processing Systems, 11955266-952999 2024b.
Chenxi Whitehouse, Tianlu Wang, Ping Yu, Xian Li, Jason Weston, Ilia Kulikov, and Swarnadeep Saha. J1: Incentivizing thinking in llm-as-a-judge via reinforcement learning. arXiv preprint Stti::2505.03222 2025.
Zihuiwen Ye, Fraser Greenlee-Sooe Scott, Max Bartolo, Phil Blunsom, Jon Ander Campos, and Matthias Gallé. Improving reward models with synthetic critiques. arXiv preprint arXiv:2405. 2024.
Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Ilia Kulikov, Kyunghyun Cho. Dong Wang. Yuandong Tian, Jason Weston, et al. Naturalrcasoning: Reasoning in the wild with 2.8 m challenging questions. arXiv preprint 250. 2025.
Sheng Zhang, Qianchu Liu, Guanghui Qin, Tristan Naumann, and Hoifung Poon. Med-rlvr: Emerging medical reasoning from a 3b base model via reinforcement learning. arXiv preprint arXiv:2502.19655, 2025.
Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards. arXiv preprint arXiv:2505. 19590, 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt- bench and chatbot arena. Advances in neural information processing systems, 36:46595-46623, 2023.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 USE OF LLMS
We LLMs for minor editorial support, including grammar, spelling, punctuation, latex formatting and word choice corrections, with usage limited to enhancing readability and clarity.
A.2 DETAILS OF CAR-MEEICINE DATASET
The following is illustrates an example from the RaR-Medicine dataset with more details about statistics and categorical distribution of rubries listed in Table 6 and 8
RaR-Medicine Example
Question. A y0-year-old male patient weighs 65 kg with a pH of 7.05, PCO2 of 15 mmHg. HCO3 of 5 mFg/I., and base deficit of -40 mFq/I. How much sodium bicarbonate should be administered in the first 4 hours to correct his metaholic acidosis?
Reference Answer. address the severe metabolic acidosis a pH of 7.05, it crucial to approach treatment with caution to avoid complications from overcorrection. The base deficit is -40 mEq/L, which indicates significant acid-base disturbance. Typically, the formula for bicarbonate administration is:
50-year-old patient (weight kg) with
Bicarbonate required
Base Deficit(mEQ/LL) x Body Weight(kg) x 0.3
Applying the values:
mFq/L. x kg x 0.3 780 mEq
However, it is standard practice to administer a partial correction initially. aiming for safety and gradual improvement. Therefore, administering about 150 mEq of sodium bicarbonate over the first 4 hours is prudent approach. This amount aligns with safe clinical guidelines initial treatment in severe metabolic acidosis, ensuring the patient starts on the path to stabilization without risking harm from rapid correction.
Rubric (checklist and weights).
Bicarbonate Calculation (w=5): Essential Criteria: The response correctly identify and apply the formula (Base Deficit x Body Weight x 0.3) to determine the bicarbonate requirement.
Safe Dosing Recommendation (w=5): Essential Criteria:
The response must state clear recommenda- tion of administering about 150 mFq of sodium bicarbonate over the first 4 hours.
Partial Correction Justification (w=4): Important Criteria:
The response should explain that only a partial correction is administered initially to avoid complications from rapid overcorrection.
Step-by-Stop Calculation (w=3); Important Criteria: The response must detail calculation steps, showing that 40 mEq/L x 65 kg x 0,3 equals 780 mEq, before noting the adjusted dose for safety.
Base Deficit Interpretation (w=2): Optional Criteria: The response may mention that the base deficit -40 mEq/L indicates severe metabolic acidosis requiring cautious treatment.
Patient Data Accuracy (w=3): Important Criteria: The response must accurately incorporate the pa- tient's weight of 65 kg and his critical pH, PCO2, and HCO3 values into the explanation.
Avoid Overcorrection Risk (w=-1): Pitfall Criteria: Does not mention the risks associated with rapid overcorrection of metabolic acidosis if only the full calculated bicarbonate amount is administered.
Metric
Value
Total examples
266
Avg. rubrics per question
7.5
Avg. question length (words)
45.0
Table Aggregate statistics for the RaR-Medicine dataset (train and validation) dataset.
14

---

Published as a conference paper at ICLR 2026
Rubric Type
Count
Percent
Important
52,748
34.1
Essential
47,584
30.7
Optional
34,261
22.1
Pitfall
20,215
13.1
Table 7: Rubric-type distribution across all 20,166 examples.
Topics
Count
Percent
Total examples
20,166
100.0
Medical Diagnosis
10,147
50.3
Medical Treatment
3,235
16.0
Medical Knowledge
2,557
12.7
Medical Diag. and Mngmmt
2,033
10.1
Medical Biology
770
3.8
Other
428
2.1
Medical Ethics
377
1.9
Health Physics
276
1.4
Epidemiology & Pub. Health
216
1.1
General Medicine
113
0.6
Forensic Medicine
14
0.1
Table 8: Distribution of topics in the medical training and validation dataset
A.3 DETAILS OF RAR-SCIENCE DATASET
This section shows an illustrative example from the RaR-Science dataset with more details about statistics and categorical distibution of rubrics listed in Table 9 10
BaR-Science Example
Question. Determine the solubility of boric acid (HşBOa) in ethanol (CHH,OH) compared to its solu- bility in benzene (C)H)),, considering the principles of 'likes dissolve likes' and the role of Kap values. Explain your reasoning and provide examples how immulcifiers could affect the solubility of substances different solvents.
Reference Answer. Boric acid is more soluble in ethanol than in benzene.
Rubric (checklist and weights).
Correct Solubility Direction (w=5): Essential Criteria: response must clearly identify that boric acid more soluble in ethanol than in benzene.
Polarity Principle (w=5): Essential Criteria:
The answer should explain how the 'like dissolves like' principle applies by contrasting the polar nature ethanol with the non-polar character benzene.
Ksp Context (w=4): Important Criteria: The response should account for the role of Ksp values, dis- cussing their typical relevance solubility even though boric acid is covalent compound rather than ionic one.
Immulcifier Explanation (w=4): Important Criteria: The answer should explain how immulcifiers could modify solubility, providing example of their effect solvation in different solvents.
Chemical Properties (w=4): Important Criteria: The response should analyze the inherent chemical properties of boric acid and the solvents to justify the observed solubility differences.
Avoid Tonic Assumptions (w=-1): Pitfall Criteria: The answer
incorrectly assume that Ksp
values for ionic compounds directly determine solubility of covalent acid such as borie acid.
Enhanced Detail (w=2); Optional Criteria: The response may include additional examples concise explanation solvation dynamics further illustrate how solubility is influenced.
15

---

Published as a conference paper at ICLR 2026
Metric
Value
Total examples
20,625
Avg. rubrics per question
7.5
Avg. question length (words)
52.6
Table 9: Aggregate statisties for the full medical (train and validation) dataset.
Rubric Type
Count
Percent
Important
52,315
34.8
Essential
42,739
28.4
Optional
33,622
22.3
Pitfall 21,808
14.5
Table 10: Rubric-type distribution across all 20 625 examples.
A.4 TRAINING DETAILS
The training hyperparameters described in Table 12
Topics
Count
Percent
Total examples
20625
100.0
General Chemistry
3163
15.3
Quantum Mechanics
3158
15.3
Physical Chemistry
2761
13.4
Statistical Mechanics
2530
12.3
Organic Chemistry
2059
10.0
General Physics
1439
7.0
Condensed Matter Physies
1387
6.7
Genetics
1378
6.7
Molecular Biology
815
4.0
Astrophysics
409
2.0
Inorganic Chemistry
407
2.0
Analytical Chemistry
398
1.9
Electromagnetism
239
1.2
Optics
143
0.7
High Energy Physics
116
0.6
Electromagnetic Theory
105
0.5
Electromagnetics
72
0.3
Relativistic Mechanics
46
0.2
Table 11: Distribution of topics in the STEM training and validation dataset
Rubrie-based evaluation consistently yields stronger policies across all judge sizes. The most pro- nounced improvement appears with Q (+0.047), where rubric guidance lifts it from weakest to nearly matching larger models. Additionally, rubric-based scores are more tightly clus- tered (0.250-0.3 0.279) than those from Likert-only judges 0..220-0.25.. indicating improved consis- tency.
These results suggest that rubrics help smaller judges approximate high-quality supervision by breaking evaluation into interpretable, binary criteria. This structured approach mitigates scale- related limitations, enabling more reliable reward modeling even with limited-capacity evaluators.
16

---

Published as a conference paper at ICLR 2026
Hyperparameters
num rollouts per prompt
16
batch size (effective)
96
1.0
warmup_ratio
0.1
learning_rate rate Ir_scheduler. -type max length num train steps
5.0e-06
constant_with warmup
3584
300
Table 12: GRPO hyperparameter settings for Medical and Science domains.
Judge Model
RaR-Implicit
Direct-Likert
GPT-4o-mini
27.9%
25.3%
Dwen-32B-Instmee
26.2%
25.4%
Qwen- 14B-Instruce
25.0%
24.9%
Qwen-7B IB-Intrucc
26.7%
22.0%
Table 13: Judge quality comparison: rubric-based evaluation pure Likert scoring synthetic medical rubrics.
A.5 EVALUATION PROMPTS
GPQA Evaluation Prompt
Determine whether the following model response matches the ground truth answer.
## Ground truth answer##:: Option (correct_answer) or (correct__nswer_texx) ## Model Response ##: (response_texts
A response is considered correct if it's final answer is the correct option letter (A, B, C. or D), or has the correct
Please respond with only "Yes" or "No" (without quotes). Do include a rationale.
A.6 PREDEFINED STATIC RUBRICS
Predefined Statie Rubrics for RaR-Static Method
The response contains correct information without factual errors, inaccuracies, or hallucinations that could mislead the user.
The response fully answers all essential parts of the question and provides sufficient detail where needed.
The response is concise and to the point, avoiding unnecessary verbosity or repetition.
The response effectively meets user's ' practical needs, provides actionable information, and genuinely helpful for their situation.
17

---

Published as a conference paper at ICLR 2026
A.7 LLM-JUDGE PROMPTS
Prompt for RAR-IMPLICIT Method
System Prompt:
You expert evaluator. Given user prompt, generated response, and list of quality rubries, please rate the overall quality the response on scale of to hased on how well satisfies the rubrics.
Consider all rubrics holistically when determining your score. A response that violates multiple rubrics should receive lower score, while response that satisfies all rubrics should receive a higher score. Start your response with valid JSON object that starts with n-- jjson" and ends with *---" The JSON object should contain single key "rating" and the value should an integer between and 10. Example response:
json
"rating" 7
User Prompt Template:
Given the following prompt. response, and rubrics, please rate the overall quality of the response scale to based on how well satisfies rubrics.
<prompt>
(prompt)
</prompt>
<response>
(response)
</response>
<rubrics>
(rubric_list_string)
</rubrics>
Your JSON Evaluation:
Prompt for DIRECT-LIKERT Baseline
System Prompt:
You are an expert evaluator. Given a user prompt and generated response, please rate the overall quality of the response on a scale of to 10, where is very poor and 10 is excellent.
Start your response with valid JSON object that starts with .- jsoon" and ends with "--*", The JSON object should contain single key "rating" and the value should be integer between and 10. Example response:
jsonn
"rating": 8
User Prompt Template:
Given the following prompt, and response, please rate the overall quality of the response on scale of to 10.
<prompt>
(prompt)
</prompt>
<response>
(response)
</response>
Your JSON Evaluation:
18

---

Published as a conference paper at ICLR 2026
Prompt for REFERENCE-LIKERT Baseline
System Prompt:
You are expert evaluator. Given a user prompt, a reference response, and generated response. please rate the overall quality of the generated response scale of to 10 based how well compares to the reference response.
Consider factors such as accuracy, completeness, coherence, and helpfulness when comparing to the refer- ence. The reference response represents a high-quality answer that you should use as a benchmark. Start your response with valid JSON object that starts with "json" and ends with object should contain single key "rating" and the value should be integer between and 10. Example response: jjson
The JSON
"rating": 8 }
User Prompt Template: Given fallowing prompt, reference response, and generated response, please rate the overall quality of the generated response on scale of to 10 hased on how well compares to the reference.
<prompt> (prompt)
</prompt>
<reference_response>
(reference)
</referrence_respone>>
<generated_responsea
(response)
digenerated_resppnee
Your JSON Evaluation:
A.8 SYNTHETIC PREFERENCE SET GENERATION
We leverage the publicly-released HEALTHBENCH (Arora al. 2025) corpus, which contains 5,000 health-related prompts accompanied by expert-written answers. Of these, 4,203 datapoints already include an ideal completion vetted by licensed clinicians. For every such prompt-ideal pair we automatically generate perturbed counterpart using 03 with the structured template shown below. The template forces the model to (i) spell out a [reasoning] plan for degrading quality, (ii) emit the degraded [perturbed completion]. and (iii) log exact [chunks added] and [chunks removed]. Perturbations are accepted only after manual screening confirms that they are objectivelyy worse, along at least one axis of medical accuracy, completeness, clarity, safety, specificity, structure, or tone, while remaining coherent and free of dangerous advice. We further exclude the prompts from HealthBench-11 IK used for ablations. This procedure produces balanced evaluation set of 3,027 preferred and 3,027 perturbed responses (6,054 total), which we use in the rubric-versus-Liker experiments in Section The prompt used for this generation is detailed in Figure A.10
A.9 JUDGE QUALITY IMPACTS ON POST-TRAINING
We assess whether rubric- rubric-guided evaluation improves judge effectiveness compared to rubric-free Likert scoring when used for GRPO training. 13]leort judge accuracy on synthetic medical data, with all policies trained using Qwen2. 5-7B and varying judge models.
19

---

Published as a conference paper at ICLR 2026
Prompt for Synthetic Rubrics Generation: Medical Domain
You are an expert rubric writer. Your job is to generate a self-contained set of evaluation criteria ("rubrics") for judging how good response is to given question. Rubrics can cover aspects of response such as, but not limited to. factual correctness, rdeal-response characteristics, style, completeness, helpfulness, harmlessness, patient-conterednesss, depth reasoning. contextual relevance, and empathy. Each item must be self-contained non expert readers should not need to infer anything or consult external information. Begin each description with its category: "Essential Criteria: "Important Criteria: "Optional Criteria: "Pitfall Criteria: Does not mention....
Inpuls:
question: The full question text.
reference answer: The ideal answer, including any specific facts, explanations, advice.
Total items:
Choose 7-20 rubric items based the complexity of the question.
Each rubric item:
title (2-4 words).
deseription:
One sentence starting with its category prefix that explicitly states exactly what to look for. For example:
Essential Criteria: Identifies non-contrast helical CT sean as most sensitive modality for ureteric stones.
Pitfall Criteria: Does not mention identifying (B) as the correct answer.
Important Criteria: Explains that non-contrast helical CT detects stones varying sizes and compositions.
Optional Criteria: States "The final answer (B)" or similar answer choice formatting.
weight: For Essenta//ttppottn//ppionnn use 1-5 (5 most important); for Pitfall, use -1 or -2.
Category guidance:
Essential: Critical facts safety checks; if missing, the response is invalid (weight 5).
Important: Key reasoning, completeness, or clarity; strongly affects quality (weight 3-4).
Optional: Helpful style extra depth; nice to have but not deal-breaking (weight 1-2).
Pitfall: Common mistakes or omissions specific to this prompt-identin things a respondent often forgets or misstates. Each Pitfall description must begin with "Pitfall Criteria: Does not mention.. or "Pitfall Criteria: Recommends... and use weight -1 or -2.
ensure self-contained guidance:
When referring answer choices, explicitly say "Identifies (A)", "Identifies (B)", etc., rather than vague phrasing.
If the format requires a conclusion like "The final answer is (B)", include a rubric item such as:
Essential Criteria: Includes clear statement "The final answer is (B)".
If reasoning should precede the answer, include rubric like:
Important Criteria: Presents the explanation before stating the final answer.
If brevity valued, include a rubric like:
Optional Criteria: Remains concise and avoids unnecessary detail.
If the question context demands mention of specific findings, include that explicitly (e.g., "Essen- tial Criteria: Mentions that CT does not require contrast").
Output: Provide JSON array of rubric objects. Each object must contain exactly three keys-tille, title, scription, and weight. Do not copy large blocks of the question or reference answer into the text. Each description must begin with its category prefix, and no extra keys are allowed. Now, given the question and reference aenseer answer, generate the rubric described. The reference answer is ideal response not necessarily exhaustive; use only as guidance.
20

---

Published as a conference paper at ICLR 2026
Prompt for Synthetic Rubric Generation: Science Domain
You are an expert rubric writer for science questions in the domains of Biology. Physics, and Chemistry. Your job is to generate self-contained set of evaluation criteria ("rubrics") for judging how good response is given question in of these domains. Rubrics of reasoning, clarity, completeness, style, helpfulness, and common pitfalls. Each rubric item must be fully self-contained so that non-expert readers need not consult any external information. Inputs:
aspects such factual correctness, depth
question: The full question text.
reference_answer. The ideal answer, including any key facts or explanations.
Total items:
Choose 7-20 rubric items based on question complexity.
Each rubric item must include exactly three keys:
1. title (2-4 words)
description: One sentence beginning with category prefix, explicitly stating what look for. For example:
States that in the described closed system, the total mechanical energy (kinetic plus potential) before the event equals the total mechanical energy after the event. Important Criteria: Breaks down numerical energy values for each stage, demonstrating that initial kinetic energy plus initial potential energy equals final kinetic energy plus final potential energy.
Essential Criteria:
Optional Criteria: Provides concrete example, such as pendulum converting between kinetic and potential energy, to illustrate how energy shifts within the system.
Pitfall Criteria: Does not mention that frictional or air-resistance losses are assumed negli- gible when applying conservation mechanical energy.
3. weight: For Essental/Impootta///piioaaal
1-5 most important); for Pitfall, -1 -2.
Category guidance:
Essential: Critical facts or safety checks; omission invalidates the response.
Important: Key reasoning or completeness; strongly affects quality.
Optional: Nice-to-have style extra depth.
Pitfall: Common mistakes or omissions; highlight things often missed.
Format notes:
When referring to answer choices, explicitly say "Identifies (A)", "Identifies (B)", etc.
If clear conclusion required (e.g. "The final answer is (B)"), include an Essential Criteria for it.
If reasoning should precede the final answer, include an Important Criteria to that effect.
If brevity valued, include an Optional Criteria about conciseness.
Output: Provide ISON array of rubric objects. Fach object must contain exactly three keys- de-
scription, and weight. Do not copy large blocks of the question reference answer into the Each description must begin with category prefix, and no extra keys are allowed. Now, given the question and reference answer, generate the rubric as described. The reference answer is an ideal response but not necessarily exhaustive; use it only as guidance.
21

---

Published as a conference paper at ICLR 2026
A.10 PERTURBED DATASET GENERATION
Prompt for Health Response Perturbation
You are tasked with creating perturbed versions health-related responses to generate training for evaluating model robustness. Given an original health prompt and an ideal complction, you must generate a perturbed version that strictly worse than the ideal response hut still plausible.
Input Format:
[prompt]: prompt
Lideal_completion]] ideal completion
Task Instructions: For the perturbed version, follow this exact process:
[reasoning]: Explain your strategy for making the response worse. Consider these perturbation ap-
proaches:
Medical Accuracy: Introduce subtle medical inaccuracies or outdated information Completeness: Re- move crucial information or context Clarity: Make explanations less clear or more confusing Safety: Reduce appropriate medical disclaimers or caveats Specificity: Replace specific advice with vague gener- alizations Structure: Disorganize the logical flow of information Tone: Make the response less empathetic dismissive
[per wwwrbeedcomleetionn. Create modified response based on your reasoning strategy.
Ldddnks.added]: List any new text segments you added (even if they 'e incorrect or unhelpful) [chunks removedd: List any text segments you removed from the ideal response
Quality Guidelines:
Ensure Degradation: Each perturbed response must be objectively worse than the idcal response- Maintain
Plausibility: The perturbed responses should still seem like reasonable attempts (not obviously nonsensi- cal) Vary Approaches: Use different perturbation strategies for each version Track Changes Precisely: Be specific about what text was added removed Health Context Sensitivity: Be mindful that health misinformation can be harmful focus subtle degradations rather than dangerous advice
Example Perturbation Strategies:
Remove important safety warnings or medical disclaimers Add outdated medical information that was once accepted but is now known to he incorrect Remove specific dosage information or timing details - Add overly general statements that replace specific guidance Remove context about when seek professional medical help- Add confusing or contradictory information Remove step-hy-step instructions and replace with vague advice unnccessarily complex medical jargon without explanation
A.11 ADDITIONAL BENCHIMARK ANALYSIS
We provide additional results for Science domain by evaluating baselines and methods trained on RaR-Science dataset by evaluating the Chemistry, Physies and Biology subset of MMLU-Pro Wang et al. (2024b). The results are shown in Table A.11
Task
Qwen2.5-8B Base
Simple-Likert
Reference-Likert
Explicit-RaR
Implicit-Ran RaR
Chemistry
45.49%
51.15%
50.88%
54.06%
52.47%
Physies
49,50%
55.04%
51.81% %
55.58%
54.27%
Biology
64.30%
68.76%
66.39%
68.34%
68.90%
Micro Avg.
51.43%
56.77% 58.32%
54,80% 56.36%
57.94% 59.33%
56.96%
Macro Avg.
53.10%
58.55%
Table 14: MMLU-Pro (Seience subset) Regression Analysis. Performance of different reward strategies on the MMLU-Pro benchmark's Science subset.
22


Reviews:
[{"review_id": "cM4c2sDVdb", "reviewer": "Reviewer_1uK3", "summary": "This paper proposes Rubrics as Rewards (RaR), a reinforcement learning framework that extends RL with Verifiable Rewards (RLVR) to domains lacking clear correctness signals—such as medical and scientific reasoning—by leveraging structured, instance-specific rubric criteria as reward functions. The authors empirically evaluate strategies for aggregating rubric feedback and demonstrate strong improvements over conventional LLM-as-judge Likert-style baselines on HealthBench (medicine) and GPQA-Diamond (science) datasets. The paper details rubric construction, aggregation schemes, experimental outcomes, and ablation studies, highlighting the benefits of rubrics for judge consistency and alignment with human preferences.", "questions": "1 For the implicit aggregation strategy (Eq. 2, Page 3), can the authors provide a deeper technical or empirical analysis of $f_\\phi$ as an LLM judge? Specifically, how sensitive is this function to prompt design, rubric length, or underlying model drift?\n\n2 Can the authors discuss the feasibility of extending rubric design and aggregation to dialog, tool-use, or more open-ended agentic tasks, and are there preliminary results or design recommendations for such settings?\n\n3 Clarify how continuous (non-binary) rubric criteria (as alluded to on Page 3) would alter reward distributions and policy learning dynamics. Are there settings (Medicine/GPQA) where continuous-valued scores were tried, and if so, what were the findings?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1 The paper tackles a timely and important limitation of RLVR by moving beyond domains with verifiable, binary reward signals, introducing a pragmatic and interpretable alternative via structured rubrics for open-ended reasoning.\n\n2 Empirical evaluation is thorough, utilizing large and diverse medical and science benchmarks (HealthBench, GPQA-Diamond), and benchmarking several methods including Direct-Likert, Reference-Likert, and filtered SFT, ensuring a fair assessment.\n\n3 The paper offers the RaR-Medicine and RaR-Science datasets with rubric annotations, valuable resources for the community.", "weaknesses": "1 Limited Generalization and Scope: Experimental evaluation is confined to domains with relatively structured reasoning (medicine, science) and primarily question-answering tasks. The approach is not shown to generalize to less-structured or dialog/agentic settings noted in Section 9. \n\n2 Implementation Ambiguities: Category-to-weight mapping for explicit aggregation (\"Essential\": 1.0, etc.) is somewhat arbitrary, and no optimization or sensitivity analysis is provided. Table 2 suggests some robustness, but further technical depth would strengthen trust and generalizability.\n\n3 Mathematical Ambiguities: The extension from binary to continuous $c_j(x, \\hat{y})$ (as briefly suggested) is not developed; implications for policy learning, reward distributions, and judge reliability are not analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GlhjtWsRfM", "reviewer": "Reviewer_xETQ", "summary": "This paper introduces Rubrics as Rewards (RaR), an RL framework that uses rubric-based feedback as structured reward signals for policy training, extending RL beyond verifiable conditions like answer-only rewards. The authors construct instance-specific rubrics in medical and scientific fields and evaluate multiple strategies for translating rubric feedback into rewards. Experiments show that RaR achieves 31% and 7% relative improvements on HealthBench and GPQA-Diamond, respectively, and that rubric-based rewards enable more consistent alignment with human preferences as model scale increases.", "questions": "1. How are the final checkpoints selected for evaluation in the RL-based baselines and RaR? Are they the final checkpoints, or are they chosen based on a validation set? If the checkpoints are selected based on intermediate validations, provide the intermediate validation plots could help clarify the effectiveness of the method.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors propose an innovative approach that uses rubric-based rewards to train models on tasks lacking verifiable reward signals in RL.\n2. The experiments and analyses demonstrate that rubric-based rewards align well with expert human preferences and are effective when the LLM-as-a-Judge model is sufficiently capable.\n3. The authors conduct experiments in the medical and scientific domains to demonstrate the effectiveness of the proposed rubric-based reinforcement learning paradigm.", "weaknesses": "1. The authors employ large-scale, instance-level rubric labeling for all training samples and use an LLM-as-a-Judge model (GPT-4o-mini) to generate rubric-based rewards in training. While this approach incurs substantial computational costs—since each rubric evaluation consumes additional tokens—the resulting performance gains, particularly on GPQA, are marginal (only 1.1 points compared to the Reference-Likert baseline). \n\n2. This paper uses only a single model, Qwen2.5-7B, as the policy model. However, experiments on a single model cannot fully demonstrate the robustness of the proposed method. Since the labeled rubrics are policy-agnostic, incorporating them into RL training for other policy models would be straightforward and could better show the method’s comprehensiveness and robustness.\n\n3. The presentation of the paper could be improved — for example, Sections 4.1 to 4.5 contain many experimental settings that could be presented more intuitively. Using tables to summarize these details would significantly enhance the paper’s readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0nxCI5elby", "reviewer": "Reviewer_19Mf", "summary": "This paper introduces Rubrics as Rewards (RaR), a framework that extends Reinforcement Learning with Verifiable Rewards (RLVR) to domains where correctness is multifaceted and not easily verifiable by a single binary signal. The authors create two datasets, RaR-Medicine and RaR-Science, by generating rubrics from reference answers using strong LLMs. Evaluations on HealthBench (rubric-based) and GPQA-Diamond (multiple-choice) show that RaR, particularly the Implicit variant, outperforms strong baselines like Direct-Likert and Reference-Likert scoring.", "questions": "See weakness above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper successfully bridges a critical gap between RLVR and preference-based RLHF by introducing a structured, multi-criteria intermediate: the rubric. The method demonstrates consistent and significant improvements over strong baselines across two challenging domains medicine and science. The relative gains of up to 31% on HealthBench are substantial.", "weaknesses": "The entire RaR pipeline's success is contingent on the quality of the synthetically generated rubrics. While ablations show the importance of using reference answers, the inherent limitations of LLM-based generation are not fully addressed. The performance ceiling is thus tied to the rubric generator's capabilities.\nThe method requires generating a unique, detailed rubric for each of the 20k+ training instances and then using an LLM judge to evaluate 16 rollouts per prompt. This is computationally expensive and may be prohibitive for larger-scale applications without significant optimizations.\nThe generalizability of RaR to more creative, open-ended, or highly subjective tasks (e.g., essay writing, dialogue, creative design) remains an open question.\nThe two aggregation strategies (Explicit and Implicit) are static. The paper does not explore more dynamic or learned weighting schemes that could potentially adapt during training.\nWhile Implicit Aggregation performs best, it also makes the reward signal less interpretable. Did you observe any cases where the implicit judge failed to properly incorporate the rubric, and could this opaqueness be a risk in safety-critical domains like medicine?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3Tk9QEvqEj", "reviewer": "Reviewer_dV3e", "summary": "This paper presents Rubrics as Rewards (RaR) that extends reinforcement learning with verifiable rewards.  It uses LLMs to generate checklist-style rubrics for each instance from the reference answer, and uses an aggregated score from these rubrics as the final rewards.  It also proposes 2 aggregation methods: explicit (with human assigned weights) and implicit (with LLM-as-a-judge to generate a score). Experiments show that their methods can out-perform the baseline methods such as likert-based rewards on benchmarks like HealthBench and GPQA Diamond.", "questions": "1. If you do not use LLM to generate rubrics but directly use the rubrics of HealthBench, would that further improve the performance of RaR on HealthBench?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "RaR shows consitent performance gain over the baseline methods without needing further human annotation.  It well-aligns the design of HealthBench, which also evaluates the responses with its own rubrics.\n\nThis paper does some ablation studies and analysis, such evaluating RaR using human preference metric. On this metric, we can see that RaR can generate more human-preferred responses.", "weaknesses": "1. The performance improvement is marginal: The paper claims a 31% relative performance gain, which is partially true. On fig 2, not all baselines are fair comparison, and they might not be trained on HealthBench or not have access to the reference answers. Comparing RaR to reference-likert baseline, its performance gain is marginal on HealthBench (2.3 pts) and not significant on GPQA-Diamond (1.1 pts and error bars overlap).\n2. Limited experiments: This paper only experiment on HealthBench and GPQA-Diamond. The former dataset is designed specifically with rubrics, thus being a good candidate dataset. But there are many multi-choice datasets similar to GPQA, and the authors should report results on these datasets to reduce the dataset noise.\n3. The main point of this paper is that rubics-based reward is better than direct prompting. However, explicit aggragation of rubric scores cannot out-perform implicit aggregation, while the \"implicit aggregation\" can be considered as a more delicate prompt that inputs question, reference answer, and response to LLM to get an overall. It has no difference to the Likert-based LLM-as-a-judge method in nature.\n4. More baselines can be introduced, such as fine-tuning with reinforcement learning and question-agnostic rubrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)