RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards
Authors: Zhilin Wang
Jiaqi Zeng
Olivier Delalleau
Ellie Evans
Daniel Egert
Hoo-Chang Shin
Felipe Soares
Yi Dong
Oleksii Kuchaiev
Venue: ICLR 2026 Poster (2026)
Keywords: reward modeling
model alignment
inference-time control
customization
LLM post-training
Primary area: foundation or frontier models, including LLMs
TL;DR: We present an approach to bridge RL with Human Feedback and Verifiable Rewards. Our method achieves #1 on JudgeBench leaderboard and exceeds or matches DeepSeek R1 and o3-mini on Arena Hard V2,WildBench and MT Bench at <5% of their inference cost.

Abstract: Reinforcement Learning with Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR) are the main RL paradigms used in LLM post-training, each offering distinct advantages. However, RLHF struggles with interpretability and reward hacking because it relies on human judgments that usually lack explicit criteria, whereas RLVR is limited in scope by its focus on correctness-based verifiers. We propose Reinforcement Learning with Binary Flexible Feedback (RLBFF), which combines the versatility of human-driven preferences with the precision of rule-based verification, enabling reward models to capture nuanced aspects of response quality beyond mere correctness.  RLBFF extracts principles that can be answered in a binary fashion (e.g. accuracy of information: yes, or code readability: no) from natural language feedback. Such principles can then be used to ground Reward Model training as an entailment task (response satisfies or does not satisfy an arbitrary principle). We show that Reward Models trained in this manner can outperform Bradley-Terry models when matched for data and achieve top performance on RM-Bench (86.2\%) and JudgeBench (81.4\%, \#1 on leaderboard as of September 24, 2025). Additionally, users can specify principles of interest at inference time to customize the focus of our reward models, in contrast to Bradley-Terry models. Finally, we present a fully open source recipe (including data) to align Qwen3-32B using RLBFF and our Reward Model, to match or exceed the performance of o3-mini and DeepSeek R1 on general alignment benchmarks of MT-Bench, WildBench, and Arena Hard v2 (at $<5$\% of the inference cost).

Published as a conference paper at ICLR 2026
RLBFF: BINARY FLEXIBLE FEEDBACK TO BRIDGE BE- TWEEN HUMAN FEEDBACK & VERIFIABLE REWARDS
Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Ellie Evans, Daniel Egert, Hoo-Chang Shin, Felipe Soares, Yi Dong, Oleksii Kuchaiev NVIDIA
[zhilinw, jiaqiz)envidie com
ABSTRACT
Reinforcement Learning with Human Feedback (RLHF) and Reinforcement Learn- ing with Verifiable Rewards (RLVR) are the main RI. paradigms used L.LM post-training, each offering distinct advantages. However, RLHF struggles with interpretability and reward hacking because relies on human judgments that usually lack explicit criteria, whereas RLVR is limited in scope by its focus on correctness-base based verifiers. We propose Reinforcement Learning with Binary Flexible Feedback (RLBFF), which combines the versatilityy of human-driven pref- erences with the precision of rule-based verification, enabling reward models to capture nuanced aspects of response quality beyond mere correctness. RLBFF extracts principles that can be answered in binary fashion (e.g. accuracy of information: "yes", or code readability: "no") from natural language feedback. Such principles can then be used to ground Reward Model training as entail- ment task (response satisfies or does not satisfy an arbitrary principle). We show that Reward Models trained in this manner can outperform Bradley-TTerry models when matched for data and achieve top performance on RM-Bench (86.2%) and JudgeBench (81 ,4%, #1 leaderboard of September 24, 2025). Additionally, users can specify principles of interest at inference time to customize the focus reward models, in contrast Bradley-Tere models. Finally, present fully open source recipe (including data) to align Qwen3-32B using RLBFF and our Reward Model, to match or exceed the performance of 03-mini and DeepSeek R1 on general alignment benchmarks of MT-Bench, WildBench, and Arena Hard v2 (at 5% of the inference cost).
Models: wwwgingface.c.//ccccooooo/o
Table 1: Comparison of Human Feedback with Verifiable Rewards for Reinforcement Learning, alongside our proposed Binary Flexible Feedback serving as a bridge between the two. See the Introduction for rationales behind the classification into good (V) and poor (X) on various aspects.
Wide Coverage Interpretability
Precision
Recall
RLHF Human Feedback (Bai et al.
2022
X
X
RLVR Verifiable Rewards Lambert et al.
2025
x
RLBFF- Binary Flexible Feedhack (This work)
/
INTRODUCTION
Reinforcement Learning with Human Feedback RLHF (Ouyang et al. 2022 Bai al. 2022 and Reinforcement Learning with Verifiable Rewards RLVR (Lambert et al.
2025 DeepSeek
AI et al., 2025) are two popular paradigms currently used for training Large Language Models (LLMs) with Reinforcement Learning (RL). Recent open-weight general-purpose LLMs are trained with both RLHF and RLVR YYagg et al. 2025a Team et al. 2025 Gemma et al. 2025) because of their complementing strengths. While these techniques have distinct advantages, propose Reinforcement Learning with Binary Flexible Feedback RLBFF that can bridge both techniques to combine their respective benefits.

---

Published as a conference paper at ICLR 2026
Formulation To formulate RLBFF, first adopt RLVR's approach of using binary rewards: given a prompt and response, a verifier will either give the response full reward the response is correct based on the verifier in KTTO (Hthayarajh et al. 2024) where a general-domane or bad. A critical difference however lies in that this binary signal in KTO may be tied to various aspects of response quality (e.g., correctness, helpfulness, coherence, etc.) that are not explicit in the data. Owing to this difference, we find that representing the principle(s) for which response is marked as good/bad can be useful- RLVR only uses the principle of correctness while KTO does not explicitly define the principle(s), meaning that the judgment is based on an unknown combination of principles. Our proposed format is hence: given a prompt, response and principle, indicate whether the response fulfills the principle.
reward otherwise. We notice a parallel between this and the rewards used
response can be marked as either good
Motivation To arrive at this formulation, we made few design choices:
1. Why principles? In various situations, the reasons why humans like/dislike response can be due to different principles. For instance, on some sub-reddits, the most highly uptvoted comment is not necessarily the most helpful correct but the most hilarious. Conversely, the most up-voted response on BaeckFxchange-aate tends to be the most correct. Learning preferences without explicitly considering the principle behind judgments can make training less effective, since the optimization objective becomes less clear (OpenAl) 2025).
2.
Why single response instead of response pair? While response pairs to a common prompt is usual for RLHF, we find that this is unnatural for most settings where people provide textual feedback online. For instance, when humans review a restaurant or a product, it's mostly focused various aspects of itself. While some implicit comparisons this is the best Thai food place in town), explicitly comparing restaurant A to restaurant is much rarer (Keller & Kostromitina 2020). Response pairs are also prone position bias (Zengg et al. 2023).
occur (e.g.
3. Why binary (instead of Likert)? Bai et al. (022)) and Ouyang et al. 2022) found that Likert scoring (e.g. 5-point scale) for evaluating LLM response be hard to calibrate across different people, since they can have different expectations on what a score:3 response is vs. score:4. At the principle level, this is even harder because for instance very concise response can be harder to separate from a concise response: binarizing the possible options (e.g. concise vs. not concise) reduces such annotation disparities.
We compare this formulation of Binary Flexible Feedback with Human Feedback and Verifiable Rewards, as summarized in in Tab. [
Wide Coverage Human Feedback can be used for a wide range of tasks that users ask of LLMs like ChatGPT (Ouyang et al., 2022). Conversely, Verifiable Rewards are typically reserved for problems with easy-to-veriie correctness, including math problems with a single correct answer (Lambert et al., 2025), competitive coding problems similar to LeetCode (DeepSeek-AT et al.) (025) and precise instruction following to check for instance, a certain word appears n times correctly (Pyatkin et al., 2025). Binary Flexible Feedback inherits the versatility of Human Feedback since principles can include any aspect that humans value (beyond only correctness).
Interpretability Given that Human Feedback is typically in form of response A is better than response B (Bai et al. 2022; Wang et al. 2025a), a trained Bradley- model produces scores that are not globally calibrated across prompts. This means that the score of each response (e.g. -14.5) can only be used and interpreted in the context of the scores of other responses to the same prompt. In addition, such a model usually operates as a black-box. i.e., it provides no explanation as to why a response received a given score. On the other hand, Verifiable Rewards are easily interpretable (i.c. either Yes or No terms of correctness), and so Binary Flexible Feedhack.
Precision and Recall Reward Models trained with Human Feedback are known to be affected by low precision- also commonly known as Reward Hacking (Weng) 2024). Specifically, this happens when the model allocates high reward to response due to features that are not widely accepted to support response quality, including matching user's 's beliefs (Sharma et al. (023) or higher response length (Duoois et al. 2025). Verifiers, on the other hand, suffer from the opposite problem of low
2

---

Published as a conference paper at ICLR 2026
The response is mostly helpful. It resolves the issue directly, provides the cor- rected full code, and follows the user's requirements ffollos the user's requirements: yes. correctly interprets the real intent of the user and fixes the correct line. However, doesn't 't have any inline comments includes inline comments: no, especially where the update is done. could be better if the code contained inline comments to line where <<=' is replaced with '<'.
Figure 1: Example of Binary Flexible Feedback in Natural Language. Text in italics are generated principles and their fulfillment; Highlighted spans are evidence that supports the principle identifica- tion with green highlight indicating fulfillment while yellow highlights indicates non-fulfillment.
recall because they may fail to recognize correct answers, which are equivalent to the reference correct answer (e.g., 3 hours vs. 180 minutes or 0.5ı vs. 90º in geometry) (Huang et al. 2025). This means that they can overly penalize correct answers. Binary Flexible Feedback reduces reward hacking by identifying a specific principle/feature for modeling. and reduces failure to recognize equivalent correct answers by training on top of LLMs that have been pretrained to recognize such cquivalence between equally correct answers. We claborate upon its thcoretical basis in App. A
Training Reward Models using Binary Flexible Feedback To empirically test the effectiveness of Binary Flexible Feedback, we attempt to find such data without success. However, we notice that the open-source dataset HelpSteer3 SeepSteer3-Feedback (NVIDIA (225a) can be easily converted into such format. We propose a method for converting feedback in natural language into a set of Binary Flexible feedback with example shown in Fig. [ We then this data train top-performing Reward Models on R.M-Bench (Liu et al. 2025a). JudgeBench (Tan et al. (025) and PrincipleBench. new human-annotated evaluation benchmark that we introduce
measure the effectiveness of
Reward Models in adhering to specific principles. Finally, we show performance of model alignment with RLBFF, by leveraging the same principles obtained from the HelpSteer3 human feedback and our trained reward model.
Main Contributions
1. Reinforcement I. Learning with Binary Flexible Feedback, which combines the benefits of RLHF and RLVR. Reward Models trained on this technique have top performance on JudgeBench (81.4%, #1 leaderboard as of 24 Sept 2025), RM-Bench and PrincipleBench.
2. PrincipleBench, a benchmark to measure reward models' ability to follow specific principles when assigning reward, a feature not seen in prior public Reward Modeling benchmarks.
3. Fully open-source formula (including data) to align Qwen3-32B with RLBFF to match or exceed the performance of proprictary models like 03-mini and DeepSeek R. on MT-Bench, WildBench and Arena Hard v2 (at < 5% of the inference cost).
2 RELATED WORK
Binary Flexible Feedback for Safety and Math Previous work has explored binary feedback in narrow domains. For safety, Mu et al. (2024) uses rule-based checks that ask an off-the-shelf LLM whether response complies with specific policies (e.g., includes a brief apology and refusal). For math, Zhang et al. (2024) trains generative verifiers to judge whether an answer is correct. These approaches operate over a small, fixed inventory of (== 10) principles. contrast, our formulation scales to 1,000+ fine-grained principles spanning general, STEM, code, and multilingual domains, substantially broadening coverage while retaining binary evaluability.
Generative Rewards Models with Self-Generater Criteria DeepSeck-GRM (Liu et al. 2025b) and RM-RI (Chen et al. 2025) learn to predict preference rankings from open-source datasets by first synthesizing rubric criteria and then grading responses against those criteria. While these self- generated rubrics provide some grounding for preference judgments, they are not user-controllable: evaluation time, users cannot specify swap in custom principles.
3

---

Published as a conference paper at ICLR 2026
Principle-Followingg Generative Reward Models RewardAnything (Yu et al.
2025) manually
curates 200 criteria spanning Content, Structure, Logic, Tone, and Style, then uses an LLM ensemble (Claude-3.7 Sonnet, GPT-4.1, DeepSeek- V3, Gemini 2.5 Pro) to assign Likert-5 labels indicating whether each response satisfies each criterion. R3 (Anugraha et al. 2025 bootstraps rubrics by consolidating task-specific annotation criteria from 10+ datasets (e.g.. UlraFeedback) into pseudo- rubrics, leveraging the datasets' ground-truth labels for supervision. LMUnit (Saad-Falcon et al., 2024) mixes both strategies: it incorporates HelpSteer2 (Wang et al. 2024 annotation criteria plus 10 coarse, hand-crafter rubrics, and augments them with synthetically generated fine- line-grained principles from an in-house set and Prometheus dataset (Kim et al., 2024). In contrast, our approach operates over >> 1,000 distinct, fine-graine, principles derived directly from human- written feedback rather than synthetic generation and yielding much broader coverage while preserving a straightforward, binary (yes/no) scoring scheme.
3 TRAINING DATA
Downloading HelpSteer3-Feedhaade from HuggingFace (NVIDIA 2025a). This dataset contains 40,821 samples with each sample consisting of one prompt (possibly multi-turn), two responses and up to three human-writter paragraph-length (2-10 sentences or 50-250 words) textual feedback per response. This feedback was written in English by 7000+ human annotators across 80+ regions for samples in General, STEM, Code and Multilingual domains (Wang et al. 2025b).
Extracting Principles and Fulfillment We define a principle as an axis upon which a response can be evaluated in a binary fashion. Rather than pre-defining fixed set of principles, we identify principles relevant to each response. We use DeepSeck V3-0324 - the strongest open-weight, non- reasoning model at the time of our experiments since this task does not require advanced reasoning capabilities, thus substantially reducing the compute required for principle extraction. We prompt the model generate greedily in JSON output format, with additional fields for supporting text spans and yes/no/partially for whether the text span supports the principle. The format was followed in 99. 99.9% of generations and the remaining were excluded. We generate principles using zero-shot prompt template (see App.
B) as early few-shot experiments heavily biased principle distributions.
Filtering Principles Unsupported by Feedback Our initial spot-checks suggest that generating principles without supporting text spans frequently leads to hallucinations. Therefore, our final prompt (see App. B) involves asking the model to cite evidence of supporting text span from the ouman-written feedback, prior to answering the satisfaction of the principle. In such setting, the model could still generate a non-existent supporting text span or slightly paraphrase the original text span. We reject extractions where the cited span does not plausibly come from the feedback using the Rapid- Fuzz string matching library (Bachmann) 2025) with rapidfuzz. parrtaallre io (feedback, text_span) > 60, removing 2.2% of principles. Such an evidence-citation mechanism minimizes LLM hallucinations relative to synthetic principle generation approaches that are not grounded in human-written feedback. In addition, we exclude all 'helpfulness' principles (since helpfulness here refers to the global quality assessment of responses - rather than being based on a particular principle). Prior to exclusion, it accounted for 4.5% of raw principles due to an artifact within FelpSteer3-Feeedback that all human-written feedback starts with 'The response is... helpful'.
Removing Partially Fulfilled Principles While we recognize that some principles can be partially fulfilled, natural language does not offer a clean way of determining whether partial means 10%, 25%, 50%, 75% or 90%. Similarly, using extent-markers in natural language such as slightly or moderately also does not offer precision, as different annotators might have different understandings of what each word means. We thus remove the principles marked as "partially" fulfilled (only 13.8%, suggesting that a binary value is suitable for most principles). Among the remaining principles, 35.4% are no and 44.6% are yes indicating only a slight imbalance, the effect of which we study App.
Obtaining High-Prect Prec-Prrcision Consensus Principles Inspired by HelpSteer2 Wang al. 2024 we find that identifying consensus across annotators can be important to reduce outlier annotations. This is because the perspective of single annotator can be subjective, which does not necessarily reflect the consensus. However, unlike HelpSteer2 where aspect ratings have numerical values that can be easily used to calculate consensus, principles are in free text. Different annotators might word
4

---

Published as a conference paper at ICLR 2026
Haral
Jon
abbedon
1790
Figure 2: 40 most frequent words in principles, excluding stop-words. Clarity, accuracy and relevance most common, followed by long tail including comprehensiveness, readability and precision.
similar principles using different terms (c.g. conrectness accuracy vs acceuracy of information). To solve this challenge, we use Qwen-3-8B Embedding (Zhang et al. 2025). which was the highest scoring model problem. We only keep principles that have at least one principle from each of all other annotators with cosine similarity 0.8 (after spot-checking 0. 0.7/0.8/0.9 and finding that 0.8 leads to matching synonyms without requiring word-for-word matching). This step was by far the most stringent filter, as we only retained around 100 thousand principles (across 3 annotators, or 33k principles with "unique" meanings) out of an initial 1.2 million. Our filtering is intentionally high- -ieeiprciiionn and low-recall, with the expectation that some correct principles will be filtered out and retaining only 1.27 principles per feedback on average (std of 0.543). We believe that such tradeoff is helpful, since we have many principles to start with and this guards against training on mis-specified criteria.
embedding benchmark MTEB (Muennighofi al. 2023) when tackled this
Human Verification To ensure that extracted principles are faithful to the natural language feedback, we conducted small human verification exercise with volunteer human annotators, who each took less than hour. Each annotator independently annotated 126 random samples, with each sample containing natural language feedback and the corresponding extracted principle. Specifically, they asked if the principle well the yes/no answer adhered to the natural language feedback. We found the inter-rater agreement to reach 0.447 Fleiss' и (moderate agreement). The majority annotator answer for each principle agreed with the extracted principle in 88.9% of cases, suggesting the robustness of principle extraction.
Principle Distribution Out of the 33k principles, there are 1,414 unique principles. Most are single-word (55.0%; e.g., clarity), followed by three-word (35.2%); the rest are two-word (5.2%), four-word (4.1%), and five-word (0.5%), with <0.1% in 6-9 words. Among principles longer than 2 words, most are conjoined by "of" (24.9%) like clarity of guidance, "and" (8. (3.6%) or "to" (5.3%). There also exist negated principles such avoidance of repetition. Fig. 2 shows the most frequent words represented in principles, indicating a diverse set of principles along many semantic axes.
4 REWARD MODELING
4.1 EVALUATION
RM-Bench and JudgeBench Following HclpSteer3-Preferenn (Wang et al. 2025c), we use two popular benchmarks RM-Bench (Liu et al. 2024) and JudgeBench (Tan et al. 2025) for evaluation. Evaluation numbers for the baseline models are from the original papers, except in situations where
THU-KEG (2025) found discrepancy between reported numbers due to inaccurate formula used for RM-Bench or when JudgeBench numbers were unavailable.
Macro-avg across domains vs. sample-avg inaccurately reported by Yu al. 2025 Details in App.
D
5

---

Published as a conference paper at ICLR 2026
PrincipleBench As RM-Bench and JudgeBench are predominantly focused on the correctness of responses, we believe is important to evaluate Reward Models' ability to follow principles on other aspects such response clarity. There is such open-source benchmark available but we saw an opportunity to create one. Specifically, we noticed that HelpSteer3- Preference (Wang et al. (025c) collected (but did not release) human annotations several binary principles: "contains incorrect information', "key information is missing', 'misses one or more specific prompt requirement(s), "contains repetitions', 'irrelevant information', 'contains awkward phrasing formatting issue' and 'response does not follow prompt language'. We were successful in obtaining additional human annotations in the validation subset (never used for training) from these authors after reaching out.
We create PrincipleBench with re- re-dodded versions of this data (App. B) Since each binary principle has been labeee by 3 annotators, we only keep principles for each sample that all three annotators agree on, to avoid accidentally including labels that contain errors. To derive a pair of chosen-rejected responses, we also only keep samples for which one response fulfills a principle while the other does not. This benchmark contains 487 samples across General, STEM, Code (with 14 programming languages) and Multilingual (with 13 natural languages) in total, similar in size to JudgeBench. In addition, PrincipleBench contains not only single-turn prompts but also context up to 10 prior turns where only the last assistant turn is considered during evaluation. Following JudgeBench, we consider pairwise GenRM get task correct only when it correctly identifies the chosen response both when it is presented before and after the rejected one, in order to minimize position bias. We report sample-level micro- micro-average as the Overall score, inspired by JudgeBench.
4.2 BASELINES
Scalar Reward Models We use Bradley-Terry Reward Model trained with similar preference data (i.e. same prompts and responses) from Wang et al.) (2025c) as a baseline. In addition, we want to explore how the Binary Flexible Feedback would work with a single fixed principle. Therefore, we used HelpSteer3-Feedhaar were marked by all three annotators as perfectly/mostly helpful (labeled Yes), or slightly/not helpful (labeled No). Prompt Templates and training details are in App. [ and respectively In addition, we also use off-the-shelf Reward Models with similar model size as baselines, including Llama-3.3- Nemotron-70B-PR Ror-wwa and Llama-3. I-Nemotron- -00----arr---
to formulate the principle of 'helpfulness' based on whether responses
trained with similar data (i.e. same prompts and responses) from Wang et al. (025c) as baseline. In addition, we use RewardAnytthnn--B--a
Generative Reward Models We use Llama-3. 3.3ma--33--enmee -.e-ooooo -ee-otron-SSpee--4gee
(Yu et al. 2025). RM-RI- 2025) and R3 B0--WWEN---4B-LOOAA-A
(Chen et al.)
(Anugraha et al. 2025) since they are the strongest models from each of these related work. We ran inference on each model following the recommended hyper-parameters and prompt template on the HuggingFace Model page and papers. We also planned to run LMUnni (Saad-Falcon et al., 2024) models but the HF model pages (ContextualAi 2025a b) were placeholders (i.e. empty) and we did not have sufficient information to infer with these models.
4.3 TRAINING
Scalar Reward Models We train Scalar RMs that predict reward scores with single generated token equivalent of compute. Following Wang al. (2025c), we start with L.ama-3.3-70B- Instruct as the base mode and train it to predict either Yes or No, given a prompt, response and principle. At evaluation time, we calculate the reward as the difference between the log-prob of Yes and the log-prob of No, conditioned upon a prompt, response and principle that best describes the evaluation sub-category. This evaluation approach is inspired by Zhang et al. (2024) and Kadavath et al. (2022). but we added support for flexible principles while prior works only supported one fixed principle. The advantage of this method lies in extreme efficiency requiring only generated token of compute during inference, whilst also providing an estimation of the confidence in fulfilling a principle, beyond getting only 'Yes' or 'No'. Prompt templates and training details are in App. B and resppettivel. Since this model can be flexibly used with any principle, we refer to it as Flexible Principles hereafter.
Yu et al. 2025) proposed R RABench, which is similar but is M- and not publicly available yet. An initial experiment with smaller 1lama-3.1-88 1-88--1--nssuuee as the base model can he found in App. G

---

Published as a conference paper at ICLR 2026
Generative Reward Models We follow Wang al. (2025c) ) to train Generative RMs (GenRMs) with reinforcement learning using the GRPO (Group Relative Policy Optimization) algorithm (Shao at. 2024). Starting with Qwen3-32B (Yang al. 2025b), we provide the model with conversation history and a principle, and train it to first reason through the task and principle, then give a final judgment Yes or No. Similar to scalar reward models, we define the reward the difference between the log-prob of Yes and the log-prob of No, and use it during both training and evaluation. Equipped with reasoning capabilities, GenRMs are designed for more complex tasks that require step-by-step reasoning before assigning rewards. On the other hand, the step-by-ste reasoning also means that GenRMs are substantially (== 2 orders of magnitude) slower and more compute-intensiv intensive during both training and inference. Given this constraint, we only train a GenRM with the best data configuration found after training various Scalar RMs. Training details and prompt templates are App. and
4.4 RESULTS
Table 2: Performance of Reward Models on R.M-Bench and JudgeBench. Higher is better.
IudgeBench Math
Model
Safety
Eacy
Knowl
Cading
Overall
Scalar Re (60.1 secord/taek)
Owrs
Flzxilec Princiles SusbuRM
74.0
Tony
N2.7
66.1
91.4
89.2
$2.0
64.2
78.5
63.0
69.4
82.1
68.9
Extemel Buselines
1ami-1 N-Nemttnn 708-Rmem
69.3
70.8
Liama Nemotron Reward
57.4
90.3
90.5
76.4
70.7
62.3
72.5
76.8
66.9
Conorative RMo 1-1C socondo /taok)
Owr
Flesible Principles GenRM
92.0
77.0
95.5
53.9
56.4
86.2
74.6
85.7
85.7
$1.4
Euernal Busefines
3.3 Nemotron Super GenRM
ResthnAnyttingg
DeepSeek- Distilldd K1-QWBNN--1G--LORRA
Flexible Principles are the top performing model across Scalar and Generative RMs As shown in Tab. 2 and 3 the Flexible Principles model is top among Scalar RMs across RM-Bench (83.6), JudgeBench (7.3) and PrincipleBench (91.6). As Generative RM experiments are computationally intensive (see App. C). we only apply it our best rest-performing ScalarRM recipe (Flexible Principles). Our Flexible Principles GenRM improves upon the Flexible Principles ScalarRM to achieve a further improvement R.M-Becc (86.2) and JudgeBench (81.4) which higher than all baseline GenRMs as well as the top of JudgeBench leaderboard (80.9) (Tan et al. 2025) as of 24 Sep 2025.
Poor Performance of Baseline GenRMs on JudgeBench We notice that while many Base- line GenRMs do well on RM-Bench, their performance were poor on JudgeBench, often under- performing Scalar RMs. For instance, Tab. shows that RewardAnything-SPP on JudgeBench, which is below the lowest performing Scalar RMM at 66.9) despite performing much better on RM-Bench. To better understand why this is so, we attempt experiment where instead of requiring both orders (chosen-fin first and reject-first) to get the answer right, we separately require only chosen-first, or ii rejected-first We find that the model performs well in chosen-fire first order at 77.1 which drops to 65.1 for rejected-first order, and further to 62.6 when both orders need to agree. This suggests substantial position bias for pairwise (or n-wise) Generative RMs, while our design of rating responses individually averts this bias, leading a SOTA performance on JudgeBench.
achieved only 62.6
Flexible Principles is the first Scalar RM enable grounding by user-specified principles While RewardAnything-88t.
and R3-QWEN3-14B- 14--LOAAKKK enabled grounding by user-specified principles previously, both of them are reasoning GenRMs that requires thousands of generated tokens per sample at inference time (while our Scalar RM only requires I generated token per sample). This means that our Flexible Principles Scalar RM can finish each task <0.11 second while GenRMs take ~100x longer- while achieving similar better benchmark performance across R.M-ench,, JudgeBench and PrincipleBench. This makes the Flexible Principles Scalar RM ideal for latency-sensitive settings that require users custom principles for scoring.
7

---

Published as a conference paper at ICLR 2026
Table 3: Performance of Reward Models on PrincipleBench. Higher is better for each category.
Aupedta Aliezan
Dasmann Scu/Maahh
Overall Average
Idfo
Compatt
Avarage
Claz
Average
Goslar Rile (<0.1 recond/tazks;
Oary
PaxiMe Prlcdlles ScelaRM
988
87.5
500
59.0
BroflyyTee Teiy
$1.4
91.5
89.8
85.7
833
88.8
90.7
88.3
90.4
87.5
88.2
90.5
89.3
99.5
Baseloos
Llam-33 Samonn lit-
-tt--ea Roward
81.5
9644
91.2
99.3
663
80.9
91.0
67.1
92.8
87.5
99.6
$4.2
98.5
89.2
Llam-31 Namomas 16S KG-Rowaad
913
$1.9
91.5
99.7
66.7
87.6
90.7
86.0
9000
R3.4
66.9
$1.1
96.3
87.5
Serw calive NKo 1910 secoedo//aad Laski
Peilte Principles GenRM
$1.4
$4.0
75.7
82.1
100
82.0
$4.0
85.2
51.3
70.8
87.4
90.5
$2.5
83.8
Entonel Beselins
Hnne33 Nomomoo Suger496 CenRM
81.4
85.7
62.5
85.8
82.3
80.5
83.3
88.1
18.9
79.3
Bewoddhnyhhngg 60kx1
76.1
61.7
63.3
81.1
87.9
90.9
15.1
76.
08.9
66.7
76.3
63.3
73.8
73.5
RMiaI-Deettiic Disitteteeeddd Q300-131
71.4
644
41.1
500
76.8
37.3
71.
19.
750
99.7
13.5
74.6
719
Poor Performance of GenRMs on PrincipleBench relative to Scalar RMs We find GenRMs generally onder-perfone Scalar RMs PrincipleBench. The highest performing GenRM, Flexible Principles GenRM at 83.8, underperforms every ScalarRM. A hypothesis is that GenRMs are typically initialized from a reasoning model (Wang et al. 2025c), which are trained excel on math, coding, and other logical reasoning benchmarks that only measure correctness of responses. Therefore, the initial reasoning process of such models over- weer-indexes on response correctness (especially logical correctness in STEM fields) and less on other aspects such as absence of repetition and response clarity, This is also supported by the higher performance of Flexible Principles GenRM on JudgeBench and RM-Bench, which concern response correctness. This highlights the value of PrincipleBench to uncover previously under-addressed aspects of response quality beyond correctness.
ABLATION STUDIES
Table Ablation of Scalar Reward Models on RM-Bench and JudgeBench. Higher better.
RM-Bench Easy
JudguBench Math
Safety
Normal
Hard
Knowl.
Resson.
Coding
Overall
Flexible Principles ScalarRM
Group Similarity00.7
82.3
82.7
69.9
96.1
81.5
83.7
83.1
82.8
70.1
71.4
H2.1
69.0
72.3
Group Similarity=-1. (defialt)
85.3
81.9
70.4
96.9
85.5
84.9
80.5
83.6
74.0
74.5
82.1
81.0
76.3
Group Similarity-419
82.9
82.2
82.7
70.8
73.5
85.7
73.7
Plexible Principles
74.5
76.3
Fived Principle Train Time
74,2
82.4
70.4
92.7
85.3
82.3
72.2
79.9
69.5
66.3
82.1
76.2
71.4
Fixod Principle Test Time
84.6
80.7
69.9
92.2
85.3
84.3
79.6
81.9
70.1
69.4
76.8
69.1
70.9
Group Similarity Threshold for Filtering Consensus Principles In Sec. 3 we chose 0.8 as the cosine similarity threshold for determining which principles to filter out. With a higher threshold, we increase the extent to which the principles across three annotators have to overlap semantically which reduces the quantity of data post- post-filtering and removes principles that are due to the subjective preference of a single annotator. This threshold has by far the most impact on data quantity, inspiring an ablation study to use 0.7 or 0.9 as the threshold. The main dataset contains 33,000 samples with the default threshold of 0.8, which increases to 95,000 for threshold of 0.7 and reduces to 11,000 for threshold of 0.9. As shown in Tab. 4 the default threshold of 0.8 achieves the best R.-BBench and JudgeBench performance indicating that the trade-off between data quantity and quality is optimal at this threshold.
Fixing Principle to Accuracy of Information at Test Time We experimented with using the Flexible Principles ScalarRM but with Accuracy of Information fixed as the principle at Test Time. Un- surprisingly, this model performed substantially worse compared to the Flexible Principle ScalarRM. However, relative to a model that was only trained on a single fixed principle, the Fixed Principle Test Time model shows substantially better performance on JudgeBench (-0.5%). This suggests that training a grinciple-follwing Reward Model can be bene- ficial even for users which only want previous findings that training on multiple tasks does not hurt and can sometimes boost performance on single tasks (Wei al. 2022
RM-Bench (+2.0%) and similar performance
single principle at evaluation time. Such results echo
8

---

Published as a conference paper at ICLR 2026
6 MODEL ALIGNMENT
Beyond evaluating the reward models' intrinsic performance, we also want to understand how they can be used to better align general-purport LLMs. We conduct an alignment experiment with the Flexible Principles GenRM, the best performing model on RM-Bench and JudgeBench. A similar experiment with the Flexible Principles ScalarRM can be found in App.
Evaluation Following Wang et al. (202cc), we use MT-Bench (Zheng et al. 2023). Arena Hard (Li
et al. 2024) and WildBench (Lin al. (025) to evaluate our models. Notably, v2 that was recently released to replace Arena Hard v0.1, which was approaching saturation (>90%
Arena Hard
for top performing models). Arena Hard v2 uses different set of prompts and stronger baseline
reference model (03-mini-2025-01-31 meaning scores are generally much lower compared to the original and not directly comparable. Details in App. H
Training We conduct Reinforcement Learning training on top of the Qwen3-32B model (ang et al. 20256) using the GRPO (Group Relative Policy Optimization) algorithm (Shao et 2024 and the same dataset used to train the Flexible Principles GenRM. The actor/policy model generates multiple candidate responses without being explicitly aware of any principle, when given a conversation context that ends with a user question. The GenRM then evaluates the quality of these responses according to the specific judging principle associated with that training sample. Similar to Sec. [4.3 the actor is trained to optimize the objective (log-probs of Yes log-probs of No) as rated by the GenRM, encouraging it to generate responses that conform as closely as possible to the principles. For training details, see App. C For comparison, we use Bradley-Terry RM from Tab, 2 as baseline RM for training the same policy model.
Table 5: Performance of Aligned Models. Higher is better for each metric except cost.
MT Bench (GPT-4-Tuboo
Arema Hard v2
WilBBachh Duth Analy.
Cost
Model
(95% CD
Overall
Creative
Plan.
Info. Seek.
Coding
In/M
Outl
Qwen3 32B
9.38
44.0 +1.5
67.57
68.63
67.95
64.68
66.78
69.53
0.018
0,072
1x
RLBFY training
9.50
55.6 (1.6616611
20.33
71.73
70.73
09.37
68.96
70,94
0.018
0,072
1x
Baseline training
9.45
(1.47-11))))
67.38
68.42
68.13
65.32
66.34
68.49
0,018
0,072
Asternal Basellnes
9:26
50.00 +0.0)
71.64
72.44
74.37
65.81
(aude--77--ommr (Thinking)
(-2.0/-15)
65:45
65.04
63.08
DeepSeck
(-207+220)
61.24
66.29
59.20
68.56
61.04
Results As shown in Tab. 5 our aligned model trained with the RLBFF technique using a fully open-source recipe and open-source data achieved similar or better performance across MT-Bench, Arena Hard v2 and WildBench, as compared to OpenAl 03-mini, Anthropic Claude-3. 3laude-3.7-Sonnet (Thinking) and DeepSeek R1 (and substantially ahead of the Baseline BT-trained Qwen3-32B). Such performance is particularly impressive since our model is much cheaper to do inference with compared to RI, 03-mini and Claude-3.7-Somme cost 1.8 cents per million input tokens and 7.2 cents per million output tokens, which is 24 187 times cheaper than compared models (assuming the same input/output tokens and a 1:1 ratio between input and output). Since our ALBFF-trained model uses identical architecture as Owen3-32D and can be served at the same cost, our RI. Rr-Fa-tr model provides similar general-alignment capabilities compared R 1/03mmini/Claude- -.aauddd-------eettt (Thinking) minuscule inference cost. (< 5% of the cheapest alternative).
Based on OpenRouter (2025) in Sep 2025, Qwen3-32B only
7 CONCLUSION
We propose Reinforcement Learning with Binary Flexible Feedback to combine the advantages of RLHF and RLVR. Leveraging RLBFF, we propose recipe that utilizes open-source data to train reward models achieving state-of- -hte--rrtt-rtttt performance on RM-Bench, JudgeBench, and Princi- pleBench. PrincipleBench, curated by us, is used to further evaluate the capacity of reward models to adhere to explicitly defined principles in reward assignment. Finally, we use RLBFF and our trained reward model to align Qwen3-32B to reach comparable performance as o3-mini and DeepSeek R1 on general alignment benchmarks of MT-Bench. WildBench and Arena Hard v2, while costing <5% for inference compared those models.

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
The procedure for pre-processing data has been described in Sec. 3 For experiments relating to
Reward Modeling and Model Alignment, details are available in Sec. 4 and as well App. B
D
and
REFERENCES
David Anugraha, Zilu Tang. Lester James V. Miranda, Hanyang Zhao, Mohammad Rifqi Farhansyah, Garry Kuwanto, Demy Wijaya, and Genta Indra Winata. R3: Robust rubric--ooii reward models, 2025. URL
abs /2w.//25555. 2505. 13388
Max Bachmann. rapidituzz/rapidfuuz: Release 3.13.0, April 2025. URL htttttt///oi.org///// 5281/zenodo.1513325
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang. Tong Zhang, Hanghang Tong, and Heng Ji. Rm-rl: Reward modeling as reasoning, 2025. URL Rolhttps:: arxiv. org/ abs 2000 02387
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and lon Stoica. Chatbot arena:
An open platform for evaluating Ilms by human preference, 2024.
ContextualAI. Lmunit- Ilama3.1 3.1.3.11000 1-bb.
LMUnit-11 11 ama3. 1 70b 2025a
ContextualAI. Lmunit- Lmunit-qwen2.5-- 4w222.5.72b.
ottp:://bugiingfaaeccco/ttttttte
LMUnit MwVnnttqwen2..o 5 72b 2025b.
DeepSeek-AI, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue. Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. I. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kcxin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang. Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wengin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu. Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo. Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou. Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li. Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang, Yukun Zha, Yuting Yan, Z. Z.. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang,
10

---

Published as a conference paper at ICLR 2026
Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
URL htttps://arxi...rrr/aa//////
12948]
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2025. URL http:/ttss///aiii.ooot abs/2404 04475
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization, 2024. URL https:://taxiiiiiieeee /LEEps///ra.v.ox////3///////
01306
Team Gemma, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej. Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan- Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng. Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang. Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimezak- Pluciíiska, Harman Singh, Harsh Mchta, Harshal Tushar Lchri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman. Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy. Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan. Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Pöder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang. Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evei, Vedant Misra, Vincent Roseberry, Vlad Feinberg. Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo. Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report, 2025. htt.t::s::/// arxiv www.wawwww...aaaaa... 1
Yuzhen Huang, Weihao Zeng. Xingshan Zeng, Qi Zhu, and Junxian He. Pitfalls of rule- and model- based verifiers- case study mathematical reasoning, 2025. RRLLhttps:://ee /ttppiv.///xivvvoo org/
2bbs22055.22 22203
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Toan-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas
11

---

Published as a conference paper at ICLR 2026
Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know, 2022. URL /attps://arxiv.org/ abs/ 2207 05221
Daniel Keller and Maria Kostromitina. Characterizing non-chain restaurants' yelp star-ratings: Generalizable findings from representative sample of yelp reviews. International Journal of
Hospitality Management, 86: 86:102440, 2020, ISSN 0278-4319. doi: hutps://doi.or//10.100
1.jhm.2219.102444 URL https::/ww...cceeeeceeeccee www.sccinnediiert............e.e http://aeeeeceee..oooooceeeeeeee com/ssceenne//arrccce.e..
011/80778:3111111232
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized evaluating other language models, 2024. URL httlattts//aaxxi..oooo
abs/2405. 01535
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers in open language model post-training, 2025. URL htttps://aaxii/// http://xxxvvvoo////o 15124
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The Arena-Hard pipeline. https hardd/ April 2024.
//lmsys. wwtp///oo///2000--111111111
Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking LLMs with challenging tasks from real users in the wild. In The Thirteenth International Conference on Learning Representations, 2025. URL http///ppppppeetpp////e/pe/eeeep new. wtr.m//rummmoooooo 2o0.ruumm22dMME0X225553333
Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. Rm-bench: Benchmarking reward models of language models with subtlety and style, 2024. URL /ttps::
abs/ 20t//2400.111 16184
Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. RM-bench: Benchmarking reward models of language models with subtlety and style. The Thirleenth International Conference Learning Representations, 2025a. URL httpss////eeeeeeew.eeeee formmf
Id=QEArm@PBdd
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling, 2025b. URL htttps://axxi...tt
0ba/2504.0264
LMSys. Arena-hard-aur leaderboard.
https:///gttub..comm
ar arena-hard-an auto 2024.
Yu Meng, Mengzhou Xia, and Dangi Chen. SimPO: Simple preference optimization with a reference- free reward, 2024.
Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for language model safety, 2024.
org/ abs /btp///1.
Niklas Muennighoff, Nouamane Tazi, Loie Magne, and Nils Reimers. MTEB: Massive text embed- ding benchmark. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2014- 2037, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10.18653/ v1/2023.each 1at.022.ecllmaii...e. 148. htt.aa/aa..aaaaaaa/aaaaa.aaaaaaa
NVIDIA. wwwiiaaHHepptterr3ttte.... HelpSteer3 Feedbacck 2025a.
Nemo NVIDIA. Nemo rl: A scalable and efficient post-training library. https:///ithuut
NVIDIA-NeMo/RR 2025b. GitHub repository.
12

---

Published as a conference paper at ICLR 2026
OpenAL. Openai model spec, Apr 2025.
URL http:://toodl--ppe..ppnnaaal
2025-04-11.htm1
OpenRouter. Openrouter. httpps//geaarotterra//mmddddtttt
2025.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. Generalizing verifiable instruction following, 2025.
URL http::htttsstttttte
org/ aww./b52207 02833
Jon Saad-Falcon, Rajan Vivek, William Berrios, Nandita Shankar Naik, Matija Franklin, Bertie Vidgen, Amanpreet Singh, Douwe Kiela, and Shikib Mehri. Lmunit: Fine- tieeggrined evaluation with natural language unit tests, 2024. URL https: arxiv.iiiivv.//ii wwapxiivorggaaaee www./ogg////.//oooo//........e.t a24g122bb/2422...../// 13091
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang. Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of
mathematical reasoning in open language models, 2024. URL httpssts:://// //apxiv.////i...o/////e/ org/ /bs/
2402. 03300
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models, 2023.
Gerald Shen, Zhilin Wang, Olivier Delalleau, Jiaqi Zeng, Yi Dong, Daniel Egert, Shengyang Sun, Jimmy Zhang, Sahil Jain, Ali Taghibakhshi, Markel Sanz Ausin, Ashwath Aithal, and Oleksii Kuchaiev. NeMo Aligner: Scalable toolkit for efficient model alignment, 2024.
Sijun Tan, Siyuan Zhuang. Kyle Montgomery, William Yuan Tang, Alejandro Cuadron, Chenguang Wang, Raluca Popa, and Ion Stoica. Judgebench: A benchmark for evaluating LLM-bbsed judges. In The Thirteenth International Conference on Learning Representations, 2025. URL https: wpw.revien..ee.eeoooooo net /or.:/ff----000daaaaa/y
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengman Dong, Angang Du, Chenzhuang Du, Dikang Du Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Hongcheng Gao, Peizhong Gao, Tong Gao, Xinran Gu, Longyu Guan, Haiqing Guo, Jianhang Guo, Hao Hu, Xiaoru Hao, Tianhong He, Weiran He, Wenyang He, Chao Hong. Yangyang Hu, Zhenxing Hu, Weixiao Huang. Zhiqi Huang, Zihao Huang, Tao Jiang. Zhejun Jiang, Xinyi Jin, Yongsheng Kang, Guokun Lai, Cheng Li, Fang Li, Haoyang Li, Ming Li, Wentao Li, Yanhao Li, Yiwei Li, Zhaowei Li, Zheming I.i, Hongzhan Lin, Xiaohan Lin, Zongyu Lin, Chengyin Liu, Chenyu Liu, Hongzhang Liu, Jingyuan Liu. Junqi Liu, Liang Liu, Shaowei Liu, T. Y. Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yibo Liu, Yiping Liu, Yue Liu, Zhengying Liu, Enzhe Lu, Lijun Lu, Shengling Ma, Xinyu Ma, Yingwei Ma, Shaoguang Mao, Jie Mei, Xin Men, Yibo Miao, Siyuan Pan, Yebo Peng, Ruoyu Qin, Bowen Qu, Zeyu Shang, Lidong Shi, Shengyuan Shi, Feifan Song, Jianlin Su, Zhengyuan Su, Xinjie Sun, Flood Sung, Heyi Tang, Jiawen Tao, Qifeng Teng, Chensi Wang, Dinglu Wang, Feng Wang. Haiming Wang, Jianzhou Wang. Jiaxing Wang, Jinhong Wang, Shengjie Wang, Shuyi Wang, Yao Wang, Yejie Wang, Yiqin Wang, Yuxin Wang, Yuzhi Wang, Zhaoji Wang. Zhengtao Wang. Zhexu Wang, Chu Wei, Qianqian Wei, Wenhao Wu, Xingzhe Wu, Yuxin Wu, Chenjun Xiao, Xiaotong Xie, Weimin Xiong, Boyu Xu, Jing Xu, Jinjing Xu, L. Xu, Lin Xu, Suting Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ziyao Xu, Junjie Yan, Yuzi Yan, Xiaofei Yang, Ying Yang. Zhen Yang, Zhilin Yang. Zonghan Yang. Haotian Yao, Xingcheng Yao, Wenjie Ye, Zhuorui Ye, Bohong Yin, Longhui Yu, Enming Yuan, Hongbang Yuan, Mengjie Yuan, Haobing Zhan, Dehao Zhang, Hao Zhang, Wanlu Zhang, Xiaobin Zhang, Yangkun Zhang, Yizhi Zhang. Yongting Zhang. Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Haotian Zhao, Yikai Zhao, Huabin Zheng, Shaojie Zheng, Jianren Zhou, Xinyu Zhou, Zaida Zhou, Zhen Zhu, Weiyu Zhuang, and Xinxing Zu. Kimi k2: Open agentic intelligence. 2025. URL https://errs abs/ 2507 .20534
13

---

Published as a conference paper at ICLR 2026
THU-KEG. Rm-bench leaderboard.
https::/gittub.comm/
RM- Bench-Leaderboae 2025.
Alan Wake, Bei Chen, C. X. Lv, Chao Li, Chengen Huang, Chenglin Cai, Chujie Zheng, Daniel Cooper, Fan Zhou, Feng Hu, Ge Zhang, Guoyin Wang. Heng Ji, Howard Qiu, Jiangcheng Zhu, Jun Tian, Katherine Su, Lihuan Zhang, Liying Li, Ming Song, Mou Li, Peng Liu, Qicheng Hu, Shawn Wang, Shijun Zhou, Shiming Yang. Shiyong Li, Tianhang Zhu, Wen Xie, Wenhao Huang, Xiang He, Xiaobo Chen, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Yanpeng Li, Yongke Zhao, Yongzhen Luo, Yuchi Xu, Yuxuan Sha, Zhaodong Yan, Zhiyuan Liu, Zirui Zhang, and Zonghong Dai. Yi-lightning technical report, 2025.
URL https: //axxiv org/ abs/ / 2412.01222
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training top- pop-perforning reward models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URLhttps: htttps://openrevttttt /pppreeeeeeii...eo/e/ee wwt.eewweeeeeeeeeeee forum??
id Id=PvVKUFhalN haNy
Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. Helpsteer2-preeeeeene Complementing ratings with preferences. In
The Thirteenth International Conference on Learning Representations, 2025a. URL https
net htt.er/eee...ne.rrorm...caaaaaao
Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Daniel Egert, Ellie Evans, Hoo- Mho-Chang Shin, Felipe Soares, Yi Dong, and Oleksii Kuchaiev. HelpSteer3: Human Human--anntaater annotated feedback and edit data to empower inference-time scaling in open-ended general-domain tasks. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 25640 25662,, Vienna, Austria, July 2025b. Association for Computational Linguistics. ISBN 979-8 879-8-20 898 17--1766-1111111 doi: 10.18653/v 2kg.20255.cc111oe
1246. URL httt::://claatth...oo
1246/
Zhilin Wang, Jiaqi Zeng. Olivier Delalleau, Hoo-Chang Shin, Felipe Soares, Alexander Bukharin, Ellie Evans, Yi Dong, and Oleksii Kuchaiev. telpstter3--preferenn Open human-annotated preference data across diverse tasks and languages, 2025c. URL https h/tp:::/aaa../.ee.e org/ass
25C5.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew Dai, and Quoe V. Le. Finetuned language models are zero-shot learners, 2022. URI.
https::/arxxis org abtp////////0/ 61652
Lilian Weng, Reward hacking in reinforcement learning. iilianweng.github... Nov 2024. URL reward hacking/
https://lilianne hwtp:/ewnnngggttt..ee wttppoots//2/55555530
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su. Yichang Zhang, Yinger Zhang. Yu Wan, Yuqiong Liu, Zekun Wang. Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025a. URL htt.t:tss://// //arxivv/ org/ abs /00p///25555 C9388
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09380 2025b.
Zhuohao Yu, Jiali Zeng, Weizheng Gu, Yidong Wang. Jindong Wang. Fandong Meng. Jie Zhou, Yue Zhang, Shikun Zhang, and Wei Ye. Rewardanything: Generalizable principle-followe reward models, 2025. pet.tp::/aariiie htt. 03637
14

---

Published as a conference paper at ICLR 2026
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agar- wal. Generative verifiers: Reward modeling as next-token prediction, 2024. URL https:/arxxv. /ttp:bbb/244.115224000 2024.
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embed- ding: Advancing text embedding and reranking through foundation models, 2025, URL attps://arxiv.
abs http///./2225555555 05176
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward over- fitting and overoptimization in rlhf, 2024. URL httph:tt///t/ :/arxiv, wwtp://rgggg../aeeeeeeoeee 2ttp://40..113333.
15

---

Published as a conference paper at ICLR 2026
A ELABORATIONS ON THEORETICAL BASIS FOR RLBFF APPROACH
The reward is derived as log P(Y) - log P(N) (where Y corresponds to the model generating "Yes". and generating "No"). As a result, such bias essentially only shifts the reward up or down, In practice we usually have P(N) = P(Y) (since the model is trained to only generate "Yes" or "No"), so the reward can be written log - log((1 - p) where p = P(Y). Plotting this function of p (e.g. in this link) shows that the shape of such reward is roughly linear, except at the boundaries of large imbalance. This suggests that as long as the imbalance is not too severe, the induced bias corresponds approximately
linear offset in the reward. In addition, we use GRPO for policy training, which normalizes the rewards in such a way that such a shift in reward is approximately being canceled out (up the non-linearity of the reward function). In GRPO, the policy does not learn from examples where all generations are given the same reward (due to reward normalization). This does not happen with our reward definition since except in very rare situations different generations lead to different token probabilities in the Reward Model output, so there is always some training signal.
We also highlight few interesting properties of our reward definition:
1. It doesn't change the theoretical optimal policy (when considering the reward alone, ignoring regularization): this optimal policy is by definition the one that always generates a response maximizing the expectation that the Reward Model generates "Yes" given the response to the prompt, which is equal to P(Y) = Since our reward can be written as log - log(1 - p), which increases monotonically with p. the optimality criterion is thus preserved.
2. Compared to using either binary reward P(Y) directly, it puts more weight the updates of responses that either fully satisfy the principle (P(Y) close to 1) or completely fail at it (P(Y) close to 0), which is form of "confidence-baseed weighting. This is because of the shape of log - log(1 - p) at the boundaries, where its derivative p(I-ş takes steep values: deriving a mathematical justification is non-trivial as we also need to account for GRPO's reward normalization, but we provide an empirical validation of this intuition in toy setting below (*).
3. Consider situation where the policy only generates good responses for given prompt, i.e. p = P(Y) is distributed in some interval close to 1 across all responses, and assume that this distribution such that Elp] also the median (e.g.. uniform or normal distribution). The strict convexity of the reward function R(p) = log log 1 p) near p 1 implies E[R(p)] > R(E) ((E(p) (Jensen's 's inequality), i.c. the mean reward is higher than the reward associated to the mean across all responses. A consequence is that on average more responses will get a reward R(p) below (((( than above it: this is because due to the previous assumption that F(p) is the median of p's distribution, 50% will get reward below R Rp))) and among those above some will fall somewhere between R (E'l)) and E[R(p)). All those responses will be given negative update in GRPO (due to the reward normalization that subtracts E[R(p)] from all rewards), while the minority of best responses will be given positive update. This encourages exploitation by focusing on the minority of very best responses when the model consistently outputs good generations. Conversely, if the policy only generates bad responses (p close 0), similar reasoning relying on the concave shape of the reward function near O shows that most responses will be given a positive update, and the minority of worst responses negative update. This encourages exploration by straying away from the minority of very worst responses and moving towards more diverse, better looking options. We believe such an adaptive exploitation / exploration mechanism may be beneficial to model training (compared e.g. to using binary rewards where there would be no training signal for rewards that are all or all I for all responses, or using R(p) p which would result in balanced positive and negative updates in the settings previously described).
Empirical validation of intuition using toy setting In addition, we describe the empirical simula- tion used to justify the theoretical benefits of the approach above. We average results over 10K runs where in each run we sample 16 values of = P(Y) from a mixture of Gaussians: we first uniformly sample number of modes M between and 5 (simulating diverse response qualities), and split the
16

---

Published as a conference paper at ICLR 2026
(0, 1) interval into M bins. Each value of p is then sampled by first uniformly sampling bin index, then sampling from a Gaussian centered within this bin. The standard deviation set to the width of the bin, and any p sampled outside of the chosen bin is re-sampled. We then compute the "relative [R(x)) weight" given to each sampled after GRPO reward normalization as w(p;) Ie Σ, [R(s))* where
R(p,) is the normalized GRPO reward: this gives us an indication of how influential that sample is in the GRPO update (whether negative positive), compared to all samples. Finally, report in the table below the average weight across all simulations as a function of p.
Table Reward magnitude across probability intervals.
Reward function Ríp)
E (0, 0.01)
E (0.01, 0.1)
pe (0.1,0.9)
p (0.9, 0.99)
p (0.99, 1.0)
(ours) log
log(1 p)
3,38
2.02
0.73
2.01
3.34
1.88
1.70
0.83
1.69
1.86
with probability p. otherwise
1.01
1.00
1.00
1.00
1.01
We from Tab. that reward definition gives more weight to samples close to the boundaries (p near O or 1), compared to either using directly as reward, or sampling a binary reward based on p.
B PROMPT TEMPLATES
Extracting Principles and Fulfillment
Feedback: <feedback>
Generate a list of principles that the response is evaluated against in the feedback. For each principle, identify text span from the feedback relating to this principle and then state whether the text span suggests that the response satisfies the principle - yes//o/partiaalll dictionary in the format ("<principle 1>"; "<aupporting text "<principle 2>": "<supporting text
Return it as a ison
span>-<yes httpr///ee///////ralll/l// span>-<yen pitessno/partiallyyt
Scalar Reward Model
<conversation>
Evaluate the response to the previous prompt in terms of whether it satisfies this principle: <principle>. Only answer Yes or No.
For evaluation, here are the principles used:
PrincipleBench
Clarity: clarity of expression
Accuracy: accuracy of facts
Relevance: alignment with prompt
No Repetition: avoidance of repetition
Language Alignment: language compliance
Essential Information: completeness of essential information
Requirements Complete: adherence to prompt requirements
RM-Bench
Math: correctness of answer
Chat: accuracy of facts
Code: accuracy
17

---

Published as a conference paper at ICLR 2026
Safety- :effuee safety compliance
Rafety-Respond: compliance with prompt instructions
JudgeBench
Knowledge: accuracy
Reasoning: ethical compliance
Math: correctness of answer
Code: instruction compliance
Generative Reward Model Prompt template:
You are an expert evaluator tasked with assessing assistant responses based on specific principles. Below is multi-turn conversation between a user and assistant. Your task is to judge whether the last assistant response adheres to the specified principle. [Start of Conversation] User:
Assistant:
XXXXX
User:
XXXXX
Assistant:
XXXXX
[End of Conversation]
[Start of Principle]
[End of Principle]
Your task:
Carefully read the entire conversation.
Judge whether the assistant's final response adheres the principle above.
3. Think step by step, citing concrete evidence from the conversation.
After your reasoning, output exactly: "Final Judgment: Yes" or "Final Judgment: No".
Do NOT output anything after the line that contains the final judgment.
C TRAINING DETAILS
Scalar Reward Model We train the Bradley-Terry model with epoch the HelpSteer3 Preferrece data as BT models are known to overfit beyond 1 epoch (Zhu et al.) [224 Wang al. (0024). For fair comparison with BT models, all other Scalar Reward Models are also only trained with 1 epoch of our preprocessed dataset. For each model, searched for the best Learning Rate among (1, 3)e-6. We trained with the Adam W optimizer with 10 warm up steps and checkpoints saved every 50 steps. We use global batch size of 128 responses and Reward Models are trained with the NeMo--ligner framework (Shen et al. 2024).
sequence length of 4096. All Scalar
Generative Reward Model We train Generative RMs for 3 epochs of our preprocessed data, For each rollout batch, there are 128 prompts and 8 responses per prompt. We use temperature=1.0 for generation and train the model with batch size of 128 and a max sequence length of 8192. We use AdamW optimizer with 10 warm up steps and save checkpoints every 10 steps. For hyperparamters, we set learning rate as 2e-6 and search KL penalty among [Ie-3, le-2, le-1, 2e-1, 3e-1). The optimal KL penalty is le-2. During training and inference, we use the logprob returned by vLLM. Since vLLM only supports returning a maximum of 20 highest logprobs, if 'Yes or 'No" are not in top-20, set reward as -50. This both makes implementation easier in practice and ensures the model is better calibrated, through penalizing very low logprobs for either 'Yes' or 'No'. All Generative RM training is conducted with the NeMo-RL framework (NVIDIA) 2025b
18

---

Published as a conference paper at ICLR 2026
Model Alignment We conduct Reinforcement Learning training for 3 epochs of our preprocessed data. The actor generates responses per prompt with temperature=110 and max sequence length 10240. We use AdamW optimizer with 10 warm up steps and save checkpoints every 10 steps. Each rollout batch contains 128 prompts, and each training batch contains 128 responses. For hyperparameters, search over (1e-6, 2e-6, 3e-6) for learning rate and (0.01, 0.05) for KL. penalty. The optimal KL penalty is 0.01. All alignment experiments are conducted with the NeMo-RL framework INVIDIA 2025b).
Compute Requirements and Optimal Hyperparameters shown in Tab.
Table 7: Compute required and optimal hyperparameters for training each model, measured in H1000node-hours. Experiments are run nodes of H100-80GB SXM GPUs on internal clusters.
Model
Compute (H100 node-hours)
I.R
Step
Scalar Reward Models
Bradley-Terry
30
lc-6
350
Fixed Principle
le-6
250
Flexible Principles
24
2e-6
256
Generative Reward Models
Flexible Principles
512
2e-6
170
Aligned Models
RLBFF Actor
864
2e-6
170
D REWARD MODEL RESULTS
Yu et al. (2025) reports the RM-BENCH performance of RewardAnything-0B-- Overall 86.4. Across the domains, Chat is at 76.7, Math 90.3, Code 75.2 and Safety 90.2. Across difficulty, Easy is 89.4, Normal 85.3 and Hard 84.4.
THU-KEG 2025) found an issue with RewardAnything-SBBy that its reported overall score (86.4) not equal to the average across Chat, Math, Code and Safety (76.7 90.3 +75.2 90.2) 8..11
To better understand how 86.4 was derived, we tested our hypothesis that this was calculated as sample-level micro-average, Liu al. (2024) states that RM-Bench contains 129 Chat, 529 Math, 228 Code and 441 Safety samples.
we use the domain-avergen and calculate sample-average, we get:
(76.7 10-1294490.000 529445557 2+228-99 2+141) 86.4
529 228 441)
Therefore, we conclude that calculated using the average was intended to be [THU-KEG) 2025). To ensure fair comparison between RewardAnything- and all other models, we use the domain-level of 83.1 its overall score and also derive the accurate difficulty level scores using a similar method.
1as.-nyttttnn-----11- RM-Bench Overall Score was inaccurately samples rather than the average across domain, RM-Bench
E MODEL. ALIGNMENT EVALUATION DETAILS
Inference Following Yang al. 2025a performing inference all models using vLLM with temperature 0.6, top-p 0.95 and max sequence length of 32,768. To ensure that the generation does not go beyond the max sequence length causing out-of-memory, set max generation length 16,384. All models are evaluated with Thinking mode on.
MT-Bench Following Wang et al. (205cc) Meng et al. (004); Wang et al. (2025a), we use GPT- 4-01 20-pereww (GrT-4-Turbo) judge with numan-verifien reference answers for code, math
19

---

Published as a conference paper at ICLR 2026
and reasoning categories. MT Bench contains 8 domains (Writing, Roleplay, Extraction, STEM, Humanities, Reasoning, Math and Coding) each with 10 tasks, containing two turns per task
WildBench Following Wang et al. (2025c); Wake et al. 2025 we use WildBench score with the default GPT-40-05-13 judge. WildBench contains 1024 diverse real-world prompts (variable- turns, not necessarily first turn) relating to Creative, Planning/Reasoning, Data Analysis/Math,
Information/Avvice seeking and Coding/Debugging.
Arena Hard Arena Hard contains 500 challenging real-worid, multilingual questions from Chat- bot Arena (Zheng et al., ) with version primarily relating to Coding and Math. For Arena Hard v2, we follow the official configuration to use the Gemini-2.55PPr judge. We believe that Gemini-2.5-Pro judge is recommended over the GPT-4.1 judge because GPT-4.1 suffers from se- vere self-enhancement bias Zheng et al. 2023), as it rates models developed by OpenAl substan- tially higher than other strong models. Specificially, LMSys (024) shows that the top 7 models rated by GPT-4.1 are all from OpenAI and the next strongest model (Gemini-2.55PPe)
is rated as only slightly better than GPT-4.1-min even though Gemini 2.5-i 2.5-Pro is widely considered as much stronger ranking 1st vs 34th on Chathot Arena on 10 Sep 2025 (Chiang al., 2024). In con- trast, Gemini-2.5-Proo is much less susceptible to such seff-enhancement bias, rating 2 OpenAl models as stronger than itself (LMSys, 2024). In addition, we also do not use style control, as we found that the way it is calculated means that scores reported are not reproducible. Specifi- cally, the official repository (LMSys 2024) uses all locally coefficients for features like markdown headings and response length. Therefore, the calculated score is highly dependent on the model judgments available locally, which can differ substan- tially across users. For instance, while the official leaderboard shows deepseek-rl as 58.0 (-2.2 / +2.0) and QwQ-32B as 43.5 (-2.5 +2.1), a local re-run of the recommended script without generating any new responses python show.result.o py --judge- names gemini- --cont col- frat-feeuures markdown length shows deepseck-rl as 50.1 (-1.8 / 1+..1) and QwQ 32B as 35.6 (-1.6/ 1-..6/+1.8). +1.8). Such large differences of around points (or > 3x of standard deviation for each model) makes it challenging to ensure style controlled scores are reproducible. On the other hand, non-style controlled scores remains consistent across various environments we tried at deepseck-rl as 48.0 and QwQ-32B as 38.1.
model-judgments to calculate
F SCALARRM ALIGNMENT EXPERIMENT
Table 8: Performance of Aligned Models. Higher better for each metric except cost.
MT Beach (GPT-44 Tumbo)
Arena Hard v2
Wildllench Data Amil.
Cost
Model
(95% CD)
Overall
Crentive
Plan.
Info. Seek.
Coding
IaM
$
Liama 3.3 708 Instruct dait. Policy)
8.29
5. 00.8
52.5
553
54.1
482
54.8
51.7
0.10
0.32
1x
+ KLBFF uaining (ScalarRb))
9.17
1126-1//e11
22.2
62.4
605
54.5
61.0
59.3
0.10
0.32
1x
apt 132-4--204440-13
8.74
1226-11/611
99.3
59.1
602
57.3
58.6
60.5
23
10
Alx
As shown in Tab. g an aligned model based on 1amma-3.3-700-Instttee with the ScalarKM is substantially worse compared to the model trained with the GenRM across MT Bench, Arena Hard, and WildBench. We believe this is because using the GenRM includes a reasoning process, which is conducive to aligning reasoning models (e.g., Qwen3-32B), which are substantially stronger in these benchmarks. Among the various benchmarks, the largest gap is seen in the Arena Hard V2 benchmark (11.9% vs. 55.6% of the Qwen3 RLBFF model in Table 5), which is likely because Arena Hard V2 contains a substantial amount of math and coding problems, which benefit from reasoning before response generation. Non-reasoning baseline models (Llama-3.3-70B, Instruct and gpt-4o) perform at lower level on this benchmark.
We focused on GenRM for alignment setup because many of the popular general-domain benchmarks (MT Bench, Arena Hard, and WildBench) mainly focus on the correctness, even if they consider other aspects of the responses (e.g., stylistic suitability). Therefore, we prioritized MM-Bench and JudgeBench, which focus on the correctness of the response (at the reward model stage) as they better predict how well-aligned models perform. There are potentially certain domain-sseciin benchmarks that would align better with principles other than correctness (e.g., in character roleplay settings
20

---

Published as a conference paper at ICLR 2026
where factual correctness matters less), but want to focus domain Hemminngeeneral alignment and hence leave those as future work.
G SCALARRM EXPERIMENT WITH SMALLER BASE MODEL
Table Performance of Small ScalarRMs on RM-Bench and JudgeBench. Higher is better.
RM Bench
JudgeBench
Midel
Chat
Math
Code
Safesy
Easy
Nornal
Hand
Overall
Knowl
Reason
Muth
Coding
Overall
second/
Principles ScalarMM
Baxtey-Tarry
71.4
58.4
54.8
Slyweek- Jamann----
59.1
76.8
We observe in Tab. that the Flexible Principles ScalarRM performs better compared to the Bradley- Terry RM trained on the same data, as well as an external baseline (Skywork-Rewand -aaaard-LLaaa331--111 across Overall R.M-Bench and Overall JudgeBench.
H SCALARRM EXPERIMENT WITH BALANCED WEIGHTED DATA
Table 10: Performance comparison of weighting of "Yes" and "No" samples across each principle on RM-BBechh and JudgeBench. Higher better.
RM-Bench Easy
JudgeBench
Math
Code
Safety
Coding
Overall
Ours
Flexible Principles ScalarMM
85,3
81.9
70.4
96.9
85.5
84.9
80.5
83.6
34.0
74.8
82.1
81.0
76.3
Equal Weightng
76.2
To better understand the impact of such imbalance in model performance, we perform reweighing to ensure that the proportion for Yes and No equal across each principle. Specifically, for every principle with an imbalance, we randomly sub-sample the majority-class to match the minority-clan This results in retaining slightly under half the original data (with around 16 thousand samples). We find in Tab. that ensuring equal weighing slightly reduced performance on R.M-Bench and maintained the same performance on JudgeBench. This confirms our earlier hypothesis that the slight imbalance in distribution does not substantially impact performance.
21


Reviews:
[{"review_id": "N8ucev3j1q", "reviewer": "Reviewer_RKAF", "summary": "The paper introduces Reinforcement Learning with Binary Flexible Feedback (RLBFF) that combines the versatility of RLHF with the precision of RLVR, enabling reward models (RMs) to capture aspects of response quality beyond correctness. RLBFF extracts principles that can be answered in a binary manner from natural language feedback. They create this training data with principles from Helpsteer3-Feedback, and use it to train reward models and also align LLMs. Experimentally, they demonstrate that RMs trained using principles outperform Bradley-Terry models and other external baselines on RM-Bench, JudgeBench, and a newly proposed PrincipleBench (designed to evaluate RMs ability to follow arbitrary principles). Additionally, they also show that a Qwen model aligned using RLBFF and their RM matches or exceeds the performance of o3-mini and Deepseek-R1 on general alignment benchmarks.", "questions": "1) Could you provide alignment results obtained using a Scalar RM, and contrast its performance with that achieved using a Generative RM?\n\n2) Given the comparatively poorer performance of GenRMs compared to ScalarRMs on PrincipleBench, which was hypothesized to result from GenRMs being initialized from reasoning models that typically excel at correctness tasks, wouldn’t ScalarRMs be a better choice for alignment in scenarios requiring broadly good performance, rather than just correctness?\n\n3) Are there any results comparing RMs trained with RLBFF and Bradley–Terry models at smaller model scales (e.g., 3B, 8B)?\n\n4) Will the principle dataset created from HelpSteer3-Feedback be released? While the recipe is provided, access to the dataset itself would greatly facilitate adoption and reproducibility.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes RLBFF, which combines the benefits of RLHF and RLVR, enabling reward models to capture arbitrary aspects of response quality beyond simple correctness. RLBFF extracts principles that can be answered in a binary fashion from feedback data and uses them to train both ScalarRMs and GenerativeRMs, improving their performance over standard Bradley–Terry models and external baselines on reward model benchmarks.\n\nThe proposed Flexible Principles Scalar RM is the first scalar reward model to enable grounding by user-specified principles, allowing for rapid inference while achieving enhanced performance over baselines.\n\nThe authors also perform alignment on a Qwen model using RLBFF and a trained principle GenRM, achieving performance that matches or exceeds o3-mini, Claude-Sonnet, and DeepSeek-R1 at a fraction of the inference cost.\n\nFurthermore, the paper details the recipe used to construct the principle data from HelpSteer3-Feedback, improving reproducibility. Additionally, the authors propose PrincipleBench, a new reward model benchmark designed to evaluate the ability of RMs to follow arbitrary principles—a capability that was previously lacking.", "weaknesses": "The authors use Generative RMs in their alignment setup. However, inference with Generative RMs is more expensive than with Scalar RMs. A comparison against alignment performed using Scalar RMs would have been enlightening, as it would clarify the performance trade-off relative to the faster inference offered by Scalar RMs. Additionally, given the comparatively poorer performance of GenRMs compared to ScalarRMs on PrincipleBench, which was hypothesized to result from GenRMs being initialized from reasoning models that typically excel at correctness tasks, ScalarRMs might be a better choice for general purpose alignment.\n\nIt would also have been valuable to see whether RLBFF provides significant gains over approaches using standard Bradley–Terry RMs at smaller model scales.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JmoT3EsmrC", "reviewer": "Reviewer_CRQz", "summary": "This paper introduces Reinforcement Learning with Binary Flexible Feedback (RLBFF), aiming to bridge RL with human feedback (RLHF) and RL with verifiable rewards (RLVR) in the context of large language model alignment. The authors argue that RLHF suffers from ambiguity and reward hacking, whereas RLVR is too narrow, and propose RLBFF, which uses human-written, principle-grounded, binary feedback to make reward assignments both interpretable and precise. They extract over one thousand fine-grained principles from HelpSteer3-Feedback, train both scalar and generative reward models on this filtered data, and evaluate models on RM-Bench, JudgeBench, and the newly curated PrincipleBench. Their Scalar Reward Model supports user-specified principles at inference, runs efficiently, and achieves state-of-the-art results on several benchmarks. The authors also show how the approach can align Qwen3-32B to strong performance on major open-source alignment benchmarks with substantial cost savings.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 Clear Motivation and Formulation: The paper gives a lucid account of the deficiencies of RLHF and RLVR, motivating the need for reward models that are both interpretable and principled (see Section 1, especially Table 1).\n\n2 Methodological Innovation: The principle-grounded binary feedback format is an interesting compromise between free-form preferences and rigid verifiable criteria. The pipeline for extracting principles via LLM prompting, with grounding in supporting text spans and consensus filtering via embedding similarity, is thoughtfully designed (see Section 3 and the discussion around the filtering strategy).\n\n3 Strong Empirical Results: The Flexible Principles reward models set a new high-water mark on RM-Bench and JudgeBench (see Table 2), and further excel on PrincipleBench, which focuses on adherence to explicit principles beyond correctness. This outperforms both scalar and generative baselines, including those requiring much heavier computation.", "weaknesses": "1 Limited Theoretical Analysis of Loss Design and Convergence: While the scalar reward modeling and binary prediction setup is elegant, there is little substantive mathematical analysis of the properties of the loss, especially with respect to class imbalance (Section 4.3). For example, while the principle fulfillment distribution is only slightly imbalanced ($35.4%$ no; $64.6%$ yes), no explicit techniques are described for dealing with such bias (e.g., reweighting, calibration). This omission is relevant since reward hacking and label bias are major themes of the work. More quantitative analysis of the loss landscape, as well as calibration studies, would add rigor.\n\n2 Benchmarks and Biases: While Table 2 and Table 3 provide strong empirical evidence, most results are for benchmarks (RM-Bench, JudgeBench, PrincipleBench) that are closely related to the training data or evaluation principle. The generalizability to tasks or principles outside the extracted HelpSteer3-Feedback distribution is not explored in depth. Are there domains or principle types where the model underperforms or fails?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KxiHwVtgyS", "reviewer": "Reviewer_Mh38", "summary": "Reinforcement Learning from Binary Flexible Feedback (RLBFF) proposes to use binary feedback derived from sets of explicit, human-interpretable principles to train scalar reward models and align large language models (LLMs). It mainly focuses on converting HelpSteer3-Feedback, an open-source dataset of natural language feedback, into structured binary evaluations along principles such as clarity, accuracy, or adherence to requirements. Unlike traditional RLHF, which relies on pairwise human preferences, or RLVR, which focuses narrowly on correctness, RLBFF extracts principle–fulfillment pairs (yes/no) to enable more transparent and fine-grained reward modeling. Using these data, the authors train flexible scalar and generative reward models that outperform previous methods on RM-Bench, JudgeBench, and the newly introduced PrincipleBench, achieving top leaderboard performance. They further apply RLBFF to align Qwen3-32B, achieving results comparable to proprietary systems like o3-mini and DeepSeek R1 on general alignment benchmarks such as MT-Bench, WildBench, and Arena Hard v2, while maintaining efficiency and interpretability at less than 5% of the inference cost.", "questions": "* The Skywork-8B-v2 model seems to report higher results on RM-bench, although on limited categories. Could you provide a comparison with your models if possible?\n* In Table 5, can you include the BT baseline model in comparison with '+RLBFF'? This will help analyze whether the dataset is, or the reward model pipeline, is more critical for improving performance.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Merging RLHF and RLVR represents an important research direction, aiming to combine the interpretability and verifiability of rule-based feedback with the flexibility and broad applicability of human preference learning.\n* RLBFF reports strong empirical results, where they outperform BT models trained on similar data and other Generative RMs on both in-distribution regarding training data (the proposed PrincipleBench) and other benchmarks(RM-bench, JudgeBench). \n* The presentation is clear and easy to follow", "weaknesses": "* Extracting principles is not possible for other datasets that do not contain textual feedback. The format or quality of the textual feedback may be critical to this method's performance. This may limit the generalizability of this approach to other datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aRIKaJFB13", "reviewer": "Reviewer_Sidm", "summary": "This paper introduces Reinforcement Learning with Binary Flexible Feedback (RLBFF), a novel paradigm designed to bridge the gap between RLHF and RLVR. RLBFF combines the versatility of human-driven preferences with the precision of rule-based verification. The method shows strong performance on RM-Bench and JudgeBench.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper clearly articulates the distinct limitations of RLHF and RLVR. The proposed RLBFF is a well-motivated solution that effectively bridges these two paradigms by incorporating the flexibility of human judgment with explicit, binary, and verifiable criteria.\n\n- Reward models trained with RLBFF achieve high performance on major alignment benchmarks.", "weaknesses": "The paper is technically sound and well-executed. I find no discernible weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)