RewardAnything: Generalizable Principle-Following Reward Models

## Abstract

Abstract Reward Models, essential for guiding Large Language Model optimization, are typically trained on fixed preference datasets, resulting in rigid alignment to single, implicit preference distributions. This prevents adaptation to diverse real-world needs—from conciseness in one task to detailed explanations in another. The standard practice of collecting task-specific preference data and retraining reward models is resource-intensive, often producing biased rewards, and limits practical application.
We introduce generalizable, principle-following reward models . We propose that RMs should understand and adhere to dynamically provided natural language specifications of reward principles, similar to instruction-following in LLMs. To measure this capability, we develop RABench , a comprehensive benchmark for RMs focusing on generalization across diverse principles. Evaluations on RABench reveal poor generalization of current RMs. As a solution, we present RewardAnything , a novel RM designed and trained to explicitly follow natural language principles. We achieve SotA performance with RewardAnything in traditional RM benchmark simply by specifying a well-defined principle, and results on RABench show we excel in adapting to novel principles without retraining. Furthermore, RewardAnything integrates seamlessly with existing RLHF methods and we show by a case study on how to automatically and efficiently align LLMs with only natural language principles .

## 1 Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse tasks, yet aligning their behavior with human preferences remains a fundamental challenge [1, 2]. Reward Models (RMs), trained on human preference data, are critical for alignment techniques like Reinforcement Learning from Human Feedback (RLHF) [3, 4], acting as proxies for human preferences to guide LLMs toward better outputs.

Figure: Figure 1: An Overview of current post-training optimization paradigm. RewardAnything is our principle-following reward model that directly rewards according to natural language principles.
Refer to caption: x1.png

Despite their importance, current reward modeling approaches involves defining a preference standard as the annotation guide, collecting human-annotated preference data (typically a prompt, two different responses for the given prompt, annotated as chosen and rejected) accordingly, and training a reward model based on this data [5, 6]. This process presents two major bottlenecks:

- 1.
Limited generalization and adaptability. Since preference datasets are static, the resulting reward models may perform well on standard benchmarks but often fail to generalize to real-world applications with different value criteria. For instance, a customer service chatbot may prioritize brevity, whereas a research assistant may emphasize detail [7]. RMs should ideally produce significantly different rewards for these different scenarios given the same prompt and responses.
Adapting to a new application typically requires collecting new preference data and retraining RMs, which can be costly and hard to scale [8].
- 2.
Bias and Interpretability Challenges from Implicit Preference Learning. Current RMs, including discriminative, generative, or even those incorporating reasoning steps - typically learn from preference datasets annotated by human experts. Although these datasets are curated and annotated with principles, most of them retain only outcome-level supervision-i.e., which response is preferred-without the underlying rationale behind each choice [9]. For example, many public datasets lack documentation of their annotation criteria. Combined with human judgment variability and information loss during reward model training, this often leads models to infer implicit values through spurious correlations or heuristics, resulting in biased signals [7] (e.g., favoring longer over accurate responses). The implicit nature of such learning also limits interpretability. Without transparent reasoning, it is difficult to explain or predict model behavior. Reliable alignment thus requires not just supervision, but interpretable, principle-grounded value modeling.

To address these challenges, we propose a shift toward principle-following reward models—systems capable of dynamically adapting their reward criteria based on explicitly provided natural language principles ([Figure 1](https://arxiv.org/html/2506.03637v2#S1.F1)).
This paradigm is analogous to transformation enabled by “instruction-following” [2] in large language models, where models generalize across diverse tasks without requiring task-specific retraining. Similarly, by endowing reward models with the ability to follow arbitrary principles or criteria, it eliminates the need to train a separate reward model for each preference scenario, thereby turning RMs into flexible tools that generalize across diverse preference contexts. Our work makes the following contributions:

- •
We formally conceptualize principle-following for RMs, categorize and curate practical principles, establishing it as a crucial capability for developing adaptable, context-aware AI systems that align with varying human preferences without extensive retraining (§[3](https://arxiv.org/html/2506.03637v2#S3)).
- •
We introduce RABench, a comprehensive benchmark to evaluate how well RMs generalize to novel natural language principles, covering different domains and highlighting current limitations and providing a basis for measuring progress of principle-following abilities (§[4](https://arxiv.org/html/2506.03637v2#S4)).
- •
We develop RewardAnything, a generative RM trained with GRPO and Group Relative Preference Learning, to interpret and apply diverse preference principles with inference-time scaling. RewardAnything is designed to efficiently rank and score groups of responses, eliminating task-specific retraining while maintaining high-quality preference judgments, and also making it computationally efficient to guide RL training like PPO and GRPO (§[5](https://arxiv.org/html/2506.03637v2#S5)).

Experiments show RewardAnything’s superior adaptation and bias mitigation, from achieving SotA performance on traditional RM benchmarks, to eliminating bias by a clear principle. A case study further demonstrates how to use RewardAnything to align an LLM with natural language principles as its sole guidance for generating rewards. This method, requiring no RM retraining, achieved significant improvements in nuanced safety, helpfulness, and response quality, validating the principle-following paradigm for efficient, flexible LLM alignment.

## 2 Related Work

### 2.1 Training and Benchmarking Reward Models

Reward Models (RMs) act as proxies for human values in LLM training [4, 2], predicting human evaluations of LLM responses [10, 11, 12]. They are typically trained on human-labeled preference data (prompt-chosen-rejected trios) [13, 9, 14, 2, 15], using objectives like the Bradley-Terry (BT) loss [16] or generative methods predicting preferences in natural language [17, 18].
RM evaluation usually involves benchmarks with preference labels [5], measuring accuracy on held-out data [19, 6]. Key benchmarks like RewardBench [20] and RM-Bench [7] offer curated prompt-chosen-rejected trios, while PPE [21] links RM performance to post-RLHF real-world human preference outcomes.

However, existing RMs and benchmarks often assume homogenous preferences [22], hindering generalization across diverse contexts or values. For example, an RM might master one implicit preference (e.g., helpfulness) but fail with others (e.g., conciseness). SALMON [23] introduced “instructable reward models” guided by explicit instructions. While they released an aligned downstream model, the instructable RM weights and its synthetic training data were not available, limiting direct replication and field exploration. Their work showed general alignment but lacked systematic study or benchmarking of the RM’s adaptability to diverse, novel principles, leaving the generalization potential of principle-following RMs unexplored. Thus, a framework to develop and evaluate RMs on their ability to robustly follow diverse principles was needed—a gap our work addresses. Other recent efforts to boost RM quality with reasoning [24, 25] or verifiable signals[26] also rely on preference pairs, making pairwise evaluation for ranking multiple candidates computationally intensive, especially with added reasoning steps. Notably [24] propose to let RMs generate different principles for the given prompt and choose the best principle automatically during inference. This approach, however, does not fully account for the diversity of human preferences, where different rankings of the same prompt-chosen-rejected trio can be reasonable under different preference criteria.
These limitations highlight the need for RMs that generalize to explicit criteria.

### 2.2 Aligning And Improving LLMs with Reward Models

RMs provide guidance signals for aligning and improving LLMs. In RLHF, an RM’s scalar reward steers LLM policy optimization via algorithms like PPO [27]. Beyond general alignment, techniques like Group Relative Policy Optimization (GRPO) [28] elicit desired behaviors like reasoning from LLMs by rewarding specific responses within rollout groups.

Alignment effectiveness hinges on RM quality; RL algorithms efficiently sample and reinforce behaviors, but models are only as aligned as their reward function.
Accuracy in predicting preferences is key but not the sole determinant of a good “teacher” RM [29]. Failure to capture nuanced criteria or adapt to contexts limits the policy. Therefore, a high-quality RM must not only be accurate but also provide sufficient reward variance to enable efficient improvements [21, 30, 29].

## 3 Principle-Following Reward Modeling Paradigm

### 3.1 The Role of Principles in Reward Modeling

Traditional reward models, trained on human preference data like chosen vs. rejected responses, learn to internalize the *implicit* criteria underlying these choices, as they typically only see decision outcomes without explicit rationales. Human preferences are also inherently multifaceted; even when annotators follow guidelines, the resulting subjectivity is hard for RMs to disentangle from the outcomes alone. This can lead RMs to learn unintended biases (e.g., favoring length over accuracy, a known issue [7, 17]). Consequently, correcting biases or adapting criteria in traditional RMs often necessitates costly re-collection of preference data tailored to new desired behaviors [31].
Principle-following RMs offer a direct solution by explicitly conditioning on natural language principles—articulated evaluation criteria—to understand and apply them directly. This promotes transparent, controllable, and adaptable AI alignment, as the model is guided by specified rules rather than inferring them from aggregate preferences.

**Table 1: Categories of principles.**
| Category | Example Principles | Count |
| --- | --- | --- |
| Content | Encourage detailed responses with relevant, illustrative examples. | 29 |
| Structure | Value responses with a clear, well-organized information flow. | 53 |
| Tone | Give responses with encouraging and helpful tone higher scores. | 36 |
| Logic | Good responses should demonstrate coherent thought processes. | 35 |
| Style | Favor responses using clear, concise language without any jargons. | 47 |

To systematically explore this, we define and categorize principles. We manually curated 200 distinct principles for later training and evaluation, categorized into five fundamental aspects of text quality, as detailed in [Table 1](https://arxiv.org/html/2506.03637v2#S3.T1). This hierarchy starts with Logic, relating to reasoning and strategic flow. Content specifies information to be presented (e.g., including progressive examples). Structure defines organization and layout. Style specifies preferences for linguistic choices. Finally, Tone captures tone and emotions (e.g., balancing honesty and encouragement). Each principle was crafted to be clear, specific, and actionable.

### 3.2 Task Definition: Principle-Following Reward Models

Formally, the task of a principle-following reward model is defined as follows: Given a natural language principle $P$ (e.g., "Principle R1" or "Principle Ra" in [Figure 2](https://arxiv.org/html/2506.03637v2#S4.F2)), a prompt $Q$, and a set of $k$ candidate responses $\{X_{1},X_{2},...,X_{k}\}$ generated for $Q$, the model must learn to produce an evaluation. This evaluation, ideally through a scoring function $S(P,Q,X_{i})\rightarrow\mathbb{R}$ for each response $X_{i}$, should reflect its adherence to principle $P$. As the principle $P$ could be arbitrary specifications, or combination of criteria with priority, the model should be able to generalize on unseen principles with different level of specificity.

## 4 RABench : A Benchmark for Evaluating Principle-Following

To address the limitations of traditional reward models and rigorously evaluate the capability of RMs to adapt to explicit instructions, we introduce RABench. It is a comprehensive benchmark designed to assess reward models’ ability to adapt their evaluation criteria based on explicitly provided natural language principles.

### 4.1 Benchmark Construction: Evaluation Set Design

The construction of the RABench evaluation set begins with sourcing its core components: principles, prompts and responses. We sampled 50 distinct principles from previously curated principles specifically for benchmarking. For prompts, to ensure a challenging and diverse evaluation, we drew from the existing RewardBench dataset [20], covering various domains like general chat, reasoning tasks like math and coding, and safety related tasks. To ensure RABench specifically evaluates principle-following rather than general helpfulness or harmlessness according to RewardBench’s original criteria, we use only their prompts without the original responses or preference labels.

Once principles and prompts were sourced for the RABench evaluation set, the next step was to generate candidate responses for each pair. We employed 10 different language models from 6 distinct families (GPT, Claude, Qwen, LLaMA, DeepSeek, details in  [Appendix D](https://arxiv.org/html/2506.03637v2#A4)) to produce these responses. Each model was instructed, via a system prompt, to generate a response that attempts to follow the given principle. This process yielded a rich collection of responses exhibiting varying degrees of adherence to each principle. For ground truth judgements, including scores and ranking for each principle-prompt pair, we first utilized four state-of-the-art LLMs as independent evaluators: Claude-3.7 Sonnet, GPT-4.1, DeepSeek-V3, and Gemini 2.5 Pro. Each LLM judge was tasked with evaluating all responses for a given prompt-principle pair by: (1) assigning scores from 1-5 based on how well each response adhered to the principle, and (2) ranking all responses from best to worst.

Given the potential for divergence among these LLM judges, we then applied a consensus algorithm with Dynamic Programming (algorithm and prompts detailed in [algorithm 1](https://arxiv.org/html/2506.03637v2#algorithm1) in the Appendix) to synthesize their individual evaluations. For each prompt-principle pair, the algorithm seeks the longest subsequence of candidates from one judge’s initial ranking where every pairwise preference within that subsequence is supported by at least $K$ (e.g., 3 out of 4) judges, based on their scores. This approach aims to find a strong, agreed-upon partial order. The longest valid subsequence found across all initial judges forms the core of the consensus ranking. The resulting consensus provides a robust ground truth ($O_{gt}$ containing scores $S$ and ranking $\Pi$) for each set of responses.

### 4.2 Quality Assurance

To ensure the quality and reliability of the algorithmically determined consensus judgments for the RABench evaluation set, we conducted a rigorous human verification process. Each element in the benchmark—consisting of a principle, prompt, list of responses, and the LLM-consensus scores and ordering—was independently evaluated by two human annotators. They determined whether the consensus ranking accurately reflected adherence to the specified principle. To ensure reliability of data, we only kept elements that both annotators considered valid.
This human verification process yielded an agreement rate of 89% with Cohen’s $\kappa$ coefficient of 0.57, indicating good inter-annotator agreement on judgment validity considering difficult and subjective nature of this task. Overall, the resulting RABench comprises 1002 validated preference rankings, and since each ranking contains several responses, this is equivalent to 31806 preference pairs in traditional benchmarks.

Figure: Figure 2: An Overview of RewardAnything. Our reward model utilizes RL with inference-time scaling to achieve strong principle-following performance while maintaining computational cost.
Refer to caption: x2.png

## 5 RewardAnything : Our Approach to Principle-Following RMs

We introduce RewardAnything, a novel reward model architecture designed to interpret and follow natural language principles when generating rewards at inference time, as illustrated in [Figure 2](https://arxiv.org/html/2506.03637v2#S4.F2). In our approach, RewardAnything generates a structured output $O_{model}$ containing reasoning, scores $\hat{S}$ for each $X_{i}$, and a ranking $\hat{\Pi}$. When trained on a dataset of listwise preferences $D=\{(P_{j},Q_{j},\text{RankedList}_{j})\}$, where $\text{RankedList}_{j}$ is an ordered list of candidate responses $X_{i}$ according to $P_{j}$, the model’s output $O_{model}$ (and thereby its parsed scores $\hat{S}$ and ranking $\hat{\Pi}$) should be highly correlated with the ground truth preferences in $D$ and generalize to unseen principles. This contrasts with traditional RMs that typically learn a single, implicit preference distribution from pairwise comparisons without explicit principle guidance.

### 5.1 Group Relative Preference Learning

To train RewardAnything, we employ Group Relative Policy Optimization (GRPO) [28], a reinforcement learning (RL) algorithm designed to refine language model behavior. In this framework, our reward model acts as a policy $\pi_{\theta}$ that learns to generate an evaluation (including reasoning, scores, and rankings) for a given set of responses based on a principle. The core idea is to reinforce outputs that accurately reflect adherence to the principle, moving beyond simple supervised prediction of absolute scores. This RL approach, focusing on relative quality discrimination, fosters better generalization. GRPO objective function, variant of the PPO-style clipped surrogate objective, is:

$$ $J_{\text{GRPO}}(\theta)=\mathbb{E}_{q,\{o_{i}\}_{i=1}^{G}\sim\pi_{\text{old}}} \left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\min\left( r_{t}(\theta)\hat{A}_{i,t},\text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon) \hat{A}_{i,t}\right)\right]-\beta D_{\text{KL}}(\pi_{\theta}||\pi_{\text{ref}})$ (1) $$

where $\theta$ are the parameters of our policy $\pi_{\theta}$. The policy generates an evaluation output $o_{i}$ (response containing scores and rankings) for an input $q$ (principle, prompt, and candidate LLM responses $X_{j}$). $r_{t}(\theta)=\frac{\pi_{\theta}(y_{t}|y_{<t},q)}{\pi_{\text{old}}(y_{t}|y_{<t
},q)}$ is the probability ratio of generating token $y_{t}$ under the current policy $\pi_{\theta}$ versus the old policy $\pi_{\text{old}}$. $\hat{A}_{i,t}$ is the advantage estimate for token $y_{t}$ in output $o_{i}$, derived from our custom reward function. The $D_{\text{KL}}(\pi_{\theta}||\pi_{\text{ref}})$ term is a KL divergence regularizer against a reference policy $\pi_{\text{ref}}$ (which is $\pi_{\text{old}}$). We formulate this approach as Group Relative Preference Learning (GRPL).

### 5.2 Reward Design

The efficacy of GRPO hinges on the design of the reward function used to compute the advantage estimates $\hat{A}_{i,t}$. Our custom reward is defined for an entire evaluation output $O_{model}$ generated by the policy $\pi_{\theta}$ (where $O_{model}$ corresponds to an $o_{i}$ in Equation 1), by comparing it against the ground truth evaluation $O_{gt}$. This overall reward $r(O_{model},O_{gt})$ is a weighted sum:

$$ $r(O_{model},O_{gt})=\lambda_{f}r_{f}(O_{model})+\lambda_{a}r_{a}(O_{model},O_{ gt})$ (2) $$

where $\lambda_{f}$ and $\lambda_{a}$ are hyperparameters; in practice, we set $\lambda_{f}=0.15$ and $\lambda_{a}=0.85$.

#### Format Reward:

$r_{f}$, encourages comprehensive, well-structured, and consistent evaluation outputs. Conceptually, this can be represented as a weighted sum of scores from $N_{f}=5$ formatting criteria:

$$ $r_{f}(O_{model})=\sum_{k=1}^{N_{f}}w_{fk}\cdot C_{k}(O_{model})$ (3) $$

where $C_{k}(O_{model})$ is the sub-score for the $k$-th criterion and $w_{fk}$ its weight. These criteria are: (1) the presence and quality of explicit reasoning within designated tags (e.g., providing a bonus for substantial thought content); (2) valid JSON structure, with considerations for parsing robustness (e.g., a minor penalty if aggressive bracket extraction was needed); (3) completeness of required keys, such as “scores” and “best-to-worst” ranking; (4) model coverage, ensuring that scores are provided for most or all ground truth models; and (5) internal consistency between the assigned scores and the explicit “best-to-worst” ranking. Each contributes to encouraging outputs that are not only parsable but also thoughtfully constructed and self-consistent.

#### Accuracy Reward:

$r_{a}$, measures how well the RM’s judgments align with the ground truth consensus, moving beyond simple rank correlation to more nuanced aspects of evaluation quality. We recognized that not all ranking errors are equally detrimental. For instance, if the ground truth scores are A: 5, B: 2, C: 1, misranking A and B is a more significant error than misranking B and C, due to the larger true quality difference between A and B. Traditional metrics might not fully capture this.
Conceptually, our $r_{a}$ is a weighted sum of $N_{a}=4$ sub-metrics:

$$ $r_{a}(O_{model},O_{gt})=\sum_{j=1}^{N_{a}}w_{aj}\cdot M_{j}(O_{model},O_{gt})$ (4) $$

where $M_{j}(O_{model},O_{gt})$ is the score from the $j$-th sub-metric and $w_{aj}$ its weight. These sub-metrics $M_{j}$ include:
(1) weighted reversed-pair penalty, emphasizing major misrankings based on true score differences. This ensures that the model prioritizes correctly ordering items with large quality distinctions over minor shuffles of similarly-scored items.
(2) score distribution matching, comparing overall statistical patterns (e.g., mean, variance) between predicted and true scores. The goal is to encourage the RM to utilize the scoring scale in a manner consistent with the ground truth.
(3) partial credit for scores close to true values. This provides a denser learning signal than a strict exact-match criterion and acknowledges that predictions closer to the true score are more valuable.
(4) ranking agreement, measured by Kendall’s $\tau$ and top-k consensus. This component captures both the global ordinal similarity of the full ranking and the critical accuracy of identifying the top-performing items.
This comprehensive reward structure guides the GRPO algorithm to train RewardAnything not just to mimic scores, but to rank relative performance with a nuanced understanding of error severity. These objectives aim to measure both absolute quality and subtle relative differences, providing dense learning signals. These designs enable learning of fine-grained relative performance and promote interpretable evaluations through the explicit reasoning process.

### 5.3 Training Data Generation for RewardAnything

The training data for RewardAnythingwas generated using a methodology similar to the benchmark creation (detailed in [section 4](https://arxiv.org/html/2506.03637v2#S4)) but with strict separation to prevent data contamination. We used a distinct set of 150 principles (from our pool of 200, non-overlapping with the 50 used in RABench) and different prompts, specifically sourcing prompts from the decontaminated Skywork-Reward trainset [9]. This process was fully synthetic and did not include the human verification step applied to the RABench evaluation data, resulted in approximately 4,000 training examples equivalent to 173K preference pairs, each comprising a principle, prompt, a list of candidate responses $X_{i}$, and the corresponding consensus preference judgment (scores $S$ and ranking $\Pi$).

**Table 2: Accuracies (%) of reward models on RM-Bench, for each domain and difficulty level. Bold, underlined indicate best and second best results. Italics mean results reported from [7, 25]. Icons: \faIconlightbulb Reasoning, \faIconstar-half-alt Pointwise Scoring, \faIcongreater-than-equal Pairwise Comparison, \faIconsort-amount-down Listwise Ranking.**
| RM-Bench | Features | Chat | Math | Code | Safety | Easy | Normal | Hard | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| General Purpose LLMs |  |  |  |  |  |  |  |  |  |
| Qwen3-8B | \faIconlightbulb \faIconstar-half-alt \faIcongreater-than-equal \faIconsort-amount-down | 66.5 | 77.1 | 57.0 | 84.4 | 76.4 | 74.3 | 74.4 | 75.0 |
| DeepSeek V3 | \faIconstar-half-alt \faIcongreater-than-equal \faIconsort-amount-down | 76.3 | 65.7 | 62.2 | 88.3 | 80.4 | 73.2 | 67.3 | 73.6 |
| Gemini 2.5 Pro | \faIconlightbulb \faIconstar-half-alt \faIcongreater-than-equal \faIconsort-amount-down | 69.3 | 36.6 | 39.1 | 89.9 | 59.3 | 56.1 | 58.4 | 57.9 |
| GPT-4.1 Nano | \faIconstar-half-alt \faIcongreater-than-equal \faIconsort-amount-down | 55.6 | 56.7 | 50.9 | 72.2 | 72.1 | 59.9 | 50.3 | 60.8 |
| GPT-4.1 | \faIconstar-half-alt \faIcongreater-than-equal \faIconsort-amount-down | 79.5 | 68.1 | 67.3 | 93.1 | 85.7 | 77.0 | 69.5 | 77.4 |
| Discriminative Reward Models |  |  |  |  |  |  |  |  |  |
| Skywork-Reward-Llama-3.1-8B-v0.2 | \faIconstar-half-alt | 69.3 | 62.1 | 53.4 | 96.0 | 89.3 | 75.8 | 52.6 | 72.6 |
| FsfairX-LLaMA3-RM-v0.1 | \faIconstar-half-alt | 67.3 | 62.8 | 55.7 | 91.8 | 87.4 | 74.8 | 52.8 | 71.7 |
| GRM-Llama3.2-3B-rewardmodel-ft | \faIconstar-half-alt | 68.6 | 61.9 | 52.8 | 95.2 | 90.8 | 75.9 | 49.4 | 72.0 |
| Nemotron-340B-Reward | \faIconstar-half-alt | 71.2 | 59.8 | 59.4 | 87.5 | 81.0 | 71.4 | 56.1 | 69.5 |
| tulu-v2.5-70b-preference-mix-rm | \faIconstar-half-alt | 58.2 | 51.4 | 55.5 | 87.1 | 72.8 | 65.6 | 50.7 | 63.0 |
| Mistral-7B-instruct-Unified-Feedback | \faIconstar-half-alt | 56.5 | 58.0 | 51.7 | 86.8 | 87.1 | 67.3 | 35.3 | 63.2 |
| Generative Reward Models |  |  |  |  |  |  |  |  |  |
| SOLAR-10.7B-Instruct-v1.0 | \faIcongreater-than-equal | 78.6 | 52.3 | 49.6 | 78.9 | 57.5 | 67.6 | 69.4 | 64.8 |
| stablelm-2-12b-chat | \faIcongreater-than-equal | 67.2 | 54.9 | 51.6 | 65.2 | 69.1 | 63.5 | 46.6 | 59.7 |
| RM-R1-Qwen-Instruct-7B | \faIconlightbulb \faIcongreater-than-equal | 66.6 | 67.0 | 54.6 | 92.6 | 79.2 | 71.7 | 59.7 | 70.2 |
| RM-R1-Qwen-Instruct-32B | \faIconlightbulb \faIcongreater-than-equal | 75.3 | 80.2 | 66.8 | 93.9 | 86.3 | 80.5 | 70.4 | 79.1 |
| RM-R1-DeepSeek-Distilled-Qwen-7B | \faIconlightbulb \faIcongreater-than-equal | 64.0 | 83.9 | 56.2 | 85.3 | 75.9 | 73.1 | 68.1 | 72.4 |
| RM-R1-DeepSeek-Distilled-Qwen-32B | \faIconlightbulb \faIcongreater-than-equal | 74.2 | 91.8 | 74.1 | 95.4 | 89.5 | 85.4 | 76.7 | 83.9 |
| RewardAnything-8B (Ours) | \faIconlightbulb \faIconstar-half-alt \faIcongreater-than-equal \faIconsort-amount-down | 76.7 | 90.3 | 75.2 | 90.2 | 89.4 | 85.3 | 84.4 | 86.4 |

## 6 Experiments

Our experiments aims to answer:
(1) How effectively can we generate rewards compared to existing reward models?
(2) Which components contribute most to principle-following capabilities?
(3) Can principle-following reward models enable more flexible alignment of language models?
Detailed Experimental Setup, including datasets, models, baselines can be found in Appendix [C](https://arxiv.org/html/2506.03637v2#A3),[D](https://arxiv.org/html/2506.03637v2#A4), and [E](https://arxiv.org/html/2506.03637v2#A5).

### 6.1 Benchmarking Reward Models and Mitigating Bias

Figure: Figure 3: What makes a good principle? RM-Bench accuracies with varying Priority goals and Clarity types. Principles prioritized with structured rules on correct goals yield better results. Details in [Appendix A](https://arxiv.org/html/2506.03637v2#A1).
Refer to caption: x3.png

As RewardAnything functions as a reward model, we initially test it on RM-Bench [7], a recent and challenging benchmark known for its "hard" setting. This setting specifically probes a common reward model bias: the inability to distinguish between a factually correct, concise response and an incorrect but detailed, formatted one. Traditional reward models often fail on these "hard" instances as they implicitly learn and sometimes misinterpret the underlying reasons for preferences in the training data. For this evaluation, we provided ours with a clear principle to focus on content accuracy and factual correctness, prioritizing it over presentation style or length, and this principle is also passed to other RMs as system prompt.
The results, detailed in [Table 2](https://arxiv.org/html/2506.03637v2#S5.T2), show that we achieve State-of-the-Art Overall performance, particularly excelling on the "hard" setting, surpassing general purpose LLMs and very recent concurrent works. This not only validates our efficacy as a general reward model but also underscores a key insight: *biases inherent in preference datasets, which traditional RMs attempt to guess, can be more directly managed and mitigated by explicitly stating the desired evaluation logic through principles*.

#### What makes a good principle?

With a principle-following reward model, it’s natural to discuss what kind of principles yield better rewards and mitigate bias better. We run experiments with different levels of priority and clarity, and provide some analysis and examples in [Appendix A](https://arxiv.org/html/2506.03637v2#A1) with results shown in [Figure 3](https://arxiv.org/html/2506.03637v2#S6.F3). The simple conclusion here is *Principles with clearly defined objective priorities and structured rules are generally better.*

Next, we evaluate on our RABench, specifically designed to assess RMs on their ability to adhere to diverse, explicit principles. Given RABench’s listwise nature and the requirement for nuanced principle interpretation, we benchmark against strong general-purpose LLMs (often employed as evaluators) and leading discriminative reward models, as it became computationally infeasible to use pairwise comparison on long lists. As shown in [Table 3](https://arxiv.org/html/2506.03637v2#S6.T3), RewardAnything demonstrates principle-following capabilities comparable to very powerful models like GPT-4.1. These experiments collectively highlight that while traditional reward models are effective for fixed, implicit preference distributions (even surpassing GPT, Gemini on RM-Bench), they often struggle to adapt to explicit principles articulated outside their original training paradigm—a gap RewardAnything effectively addresses.

**Table 3: Performance of reward models on RABench. Values represent scores for each domain and principle category. For overall metrics, Accuracy is pairwise ranking accuracy (%), Kendall’s $\tau$ measures ranking correlation, NDCG evaluates ranking quality, and Var. indicates score variance. Bold indicates best performance and underlined indicates second best.**
| Model | Domains | Principle Categories | Overall |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Chat | Code | Safety | Math | Content | Logic | Tone | Style | Structure | Accuracy | Kendall’s $\tau$ | NDCG | Var. |
| Discriminative Reward Models (w/ Pointwise Scoring \faIconstar-half-alt) |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Skywork-Reward-Llama-3.1-8B-v0.2 | 64.7 | 60.3 | 67.6 | 58.6 | 63.3 | 61.1 | 64.5 | 64.5 | 62.0 | 63.0 | 26.07 | 91.03 | 123.04 |
| FsfairX-LLaMA3-RM-v0.1 | 73.6 | 64.3 | 82.6 | 66.7 | 75.7 | 68.0 | 76.1 | 70.0 | 69.8 | 71.4 | 42.90 | 93.67 | 3.03 |
| GRM-Llama3.2-3B-rewardmodel-ft | 66.1 | 62.7 | 69.8 | 58.9 | 64.3 | 62.1 | 65.5 | 66.9 | 64.5 | 64.8 | 29.54 | 91.42 | 12.03 |
| General Purpose & Generative Reward Models (w/ Listwise Ranking \faIconsort-amount-down) |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Qwen3-8B | 71.7 | 69.2 | 77.7 | 66.6 | 74.7 | 72.1 | 74.6 | 70.7 | 67.8 | 71.3 | 53.00 | 87.49 | 1.06 |
| DeepSeek-V3 | 80.5 | 79.5 | 84.3 | 79.2 | 82.8 | 79.5 | 81.9 | 80.8 | 79.9 | 80.7 | 61.49 | 96.89 | 1.17 |
| Gemini 2.5 Pro | 76.0 | 63.5 | 83.3 | 72.0 | 65.0 | 76.1 | 77.7 | 71.2 | 69.7 | 72.8 | 60.10 | 84.25 | 1.53 |
| GPT-4.1 Nano | 65.3 | 61.9 | 69.2 | 59.8 | 64.3 | 62.4 | 67.4 | 64.4 | 62.8 | 64.3 | 30.95 | 92.46 | 0.91 |
| GPT-4.1 | 82.1 | 82.4 | 83.8 | 81.8 | 86.0 | 83.8 | 81.6 | 82.2 | 81.7 | 82.5 | 64.90 | 97.18 | 1.38 |
| RewardAnything-8B | 81.6 | 81.9 | 84.4 | 79.6 | 84.2 | 82.0 | 82.2 | 82.1 | 81.1 | 81.9 | 65.27 | 97.84 | 1.46 |

### 6.2 Ablation Studies on RewardAnything

**Table 4: Ablation Studies. We analyze the impact of components on RABench accuracies.**
| Method | Chat | Code | Safety | Math | Overall |
| --- | --- | --- | --- | --- | --- |
| RewardAnything-8B | 81.6 | 81.9 | 84.4 | 79.6 | 81.9 |
| Backbone (Qwen3-8B) | 71.7 | 69.2 | 77.7 | 66.6 | 71.3 |
| Training Ablations: |  |  |  |  |  |
| - Principles | 71.4 | 57.5 | 75.6 | 68.2 | 67.4 |
| Listwise $\to$ Pairwise | 74.4 | 69.6 | 79.2 | 70.4 | 73.2 |
| GRPO $\to$ SFT | 59.0 | 64.6 | 66.4 | 60.4 | 62.3 |
| GRPO Reward Ablations: |  |  |  |  |  |
| - Relative Preference | 79.1 | 77.3 | 80.4 | 75.2 | 78.2 |
| - Format | 79.3 | 77.8 | 80.6 | 75.3 | 78.5 |
| Inference Ablation: |  |  |  |  |  |
| - Reasoning | 74.8 | 74.1 | 77.4 | 65.3 | 73.9 |

Training Ablations.
Removing principles during training and inference (“- Principles”), effectively mimicking traditional RMs that learn from preferences without explicit rationales, causes a significant degradation, highlighting the critical role of explicit principle guidance.
Converting listwise data to pairwise comparisons (“Listwise $\to$ Pairwise”) reduces performance to 73. Listwise training is more efficient and allows the model to learn finer-grained distinctions, crucial for practical RLHF where multiple candidates are scored. *We comprehensively compare the Pointwise, Pairwise and Listwise RMs, including computation cost, strengths and weaknesses in [Appendix B](https://arxiv.org/html/2506.03637v2#A2)*.
Replacing GRPO with Supervised Fine-Tuning (“GRPO $\to$ SFT”) on the same data—an approach resembling SALMON’s [23] but with more targeted principle-following data for our baseline—leads to the largest degradation. This suggests SFT alone tends to overfit and memorize, while GRPO fosters better generalization for principle-following.

GRPO Reward Ablations.
Modifying the accuracy reward to only use exact matching instead of relative preference signals (“- Relative Preference”) results in a slight drop, indicating that dense, relative signals are important for efficient learning, as exact matches are sparse.
Removing the format reward (“- Format”) has a smaller impact. While the accuracy reward implicitly requires correct format, an explicit format reward likely aids training stability and convergence speed.

Inference Ablation.
Disabling the generation of reasoning during inference (“- Reasoning”) significantly degrades performance to 73. This confirms that the explicit reasoning process is integral to RewardAnything’s ability to accurately apply principles at inference time.

### 6.3 Case Study: Aligning LLM with Natural Language Principles

We don’t just stop with high scores on benchmarks. To demonstrate RewardAnything’s practical utility, we aligned Qwen3-8B for nuanced safety behaviors. This was achieved using only a natural language principle and 2000 prompts (prompts only, no preference data or custom RM training needed) from the PKU-SafeRLHF trainset [32]. The objective was multifaceted: correctly refuse harmful requests, minimize over-rejection of safe queries, and for unsafe queries, provide gentle, alternative-offering responses rather than blunt refusals. This alignment was guided by a single, complex, task-specific Out-of-Distribution (OOD) principle (detailed in  [Figure 4](https://arxiv.org/html/2506.03637v2#S6.F4)), distinct from our 200 training/evaluation principles and incorporating prioritized, conflicting goals. Qwen3-8B was aligned via GRPO with RewardAnything as the sole reward source. For a robust comparison, we also aligned a model using Skywork RM—chosen for its leading performance on the RM-Bench Safety domain—under an identical GRPO setup, also leveraging these 2000 prompts. Evaluation employed XSTest [33] and MT-Bench [10] (for pairwise text quality, judged by GPT-4.1).

Figure: Figure 4: Example of aligning LLMs with NL principles. We produced an aligned model that offers helpful and warm responses to nuanced safety prompts, outperforming simple baseline refusals.
Refer to caption: x4.png

The results, presented in  [Figure 4](https://arxiv.org/html/2506.03637v2#S6.F4), demonstrate the effectiveness of this principle-driven alignment. Our RewardAnything-aligned model (“Ours-Aligned”) not only slightly reduced incorrect rejections for safe prompts compared to the original Qwen3-8B and the Skywork-Aligned model, but also improved handling of unsafe prompts. Specifically, it maintained a high refusal rate for unsafe content while transforming more of these necessary refusals into constructive, empathetic, and alternative-providing responses, showcasing a responsible, human-like approach. Furthermore, text quality comparisons on MT-Bench reveal that Ours-Aligned significantly outperforms both the original Qwen3-8B and the Skywork-Aligned model, indicating that this nuanced safety alignment also leads to higher overall response quality.

This case study substantiates that RewardAnything is not just an incremental improvement but facilitates a new paradigm for LLM alignment. It empowers users to flexibly and directly steer models towards complex, desired behaviors using natural language specifications, truly embodying the Reward*Anything* vision and significantly lowering the barrier to creating customized, deeply aligned AI systems.

## 7 Conclusion

In this work, we addressed the limitations of traditional reward models, particularly their struggle with homogenous preference assumptions and adapting to explicit, diverse criteria. We introduced RewardAnything, a novel principle-following reward model trained with GRPO, and RABench, a comprehensive benchmark designed to evaluate adherence to natural language principles. Our experiments demonstrate that RewardAnything not only achieves state-of-the-art performance on existing benchmarks like RM-Bench by leveraging explicit principles to mitigate bias, but also excels on RABench, showcasing strong generalization to unseen, complex principles. Ablation studies confirmed the importance of explicit principle guidance, listwise training, and reasoning. Crucially, our case study illustrated RewardAnything’s practical utility in aligning an LLM to nuanced, out-of-distribution safety behaviors using only a natural language principle and a small set of prompts, outperforming strong baselines in both safety metrics and text quality. This research paves the way for more flexible, controllable, and efficient LLM alignment, significantly lowering the barrier to creating AI systems that can be directly guided by human-specified values and instructions.

## References

- [1]
D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving, “Fine-tuning language models from human preferences,” *arXiv preprint arXiv:1909.08593*, 2019.
- [2]
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray *et al.*, “Training language models to follow instructions with human feedback,” *Advances in Neural Information Processing Systems*, vol. 35, pp. 27 730–27 744, 2022.
- [3]
P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” *Advances in neural information processing systems*, vol. 30, 2017.
- [4]
N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano, “Learning to summarize with human feedback,” *Advances in neural information processing systems*, vol. 33, pp. 3008–3021, 2020.
- [5]
A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma *et al.*, “A general language assistant as a laboratory for alignment,” *arXiv preprint arXiv:2112.00861*, 2021.
- [6]
Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan *et al.*, “Training a helpful and harmless assistant with reinforcement learning from human feedback,” *arXiv preprint arXiv:2204.05862*, 2022.
- [7]
Y. Liu, Z. Yao, R. Min, Y. Cao, L. Hou, and J. Li, “Rm-bench: Benchmarking reward models of language models with subtlety and style,” *arXiv preprint arXiv:2410.16184*, 2024.
- [8]
T. Kaufmann, P. Weng, V. Bengs, and E. Hüllermeier, “A survey of reinforcement learning from human feedback,” *arXiv preprint arXiv:2312.14925*, vol. 10, 2023.
- [9]
C. Y. Liu, L. Zeng, J. Liu, R. Yan, J. He, C. Wang, S. Yan, Y. Liu, and Y. Zhou, “Skywork-reward: Bag of tricks for reward modeling in llms,” *arXiv preprint arXiv:2410.18451*, 2024.
- [10]
L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing *et al.*, “Judging llm-as-a-judge with mt-bench and chatbot arena,” *Advances in Neural Information Processing Systems*, vol. 36, pp. 46 595–46 623, 2023.
- [11]
Z. Zeng, J. Yu, T. Gao, Y. Meng, T. Goyal, and D. Chen, “Evaluating large language models at evaluating instruction following,” *arXiv preprint arXiv:2310.07641*, 2023.
- [12]
Z. Yu, C. Gao, W. Yao, Y. Wang, W. Ye, J. Wang, X. Xie, Y. Zhang, and S. Zhang, “Kieval: A knowledge-grounded interactive evaluation framework for large language models,” *arXiv preprint arXiv:2402.15043*, 2024.
- [13]
Z. Wang, Y. Dong, O. Delalleau, J. Zeng, G. Shen, D. Egert, J. J. Zhang, M. N. Sreedhar, and O. Kuchaiev, “Helpsteer2: Open-source dataset for training top-performing reward models,” 2024.
- [14]
Y. Dubois, C. X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. S. Liang, and T. B. Hashimoto, “Alpacafarm: A simulation framework for methods that learn from human feedback,” *Advances in Neural Information Processing Systems*, vol. 36, pp. 30 039–30 069, 2023.
- [15]
H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe, “Let’s verify step by step,” in *The Twelfth International Conference on Learning Representations*, 2023.
- [16]
R. A. Bradley and M. E. Terry, “Rank analysis of incomplete block designs: I. the method of paired comparisons,” *Biometrika*, vol. 39, no. 3/4, pp. 324–345, 1952.
- [17]
Y. Wang, Z. Yu, Z. Zeng, L. Yang, C. Wang, H. Chen, C. Jiang, R. Xie, J. Wang, X. Xie *et al.*, “Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization,” *arXiv preprint arXiv:2306.05087*, 2023.
- [18]
S. Kim, J. Shin, Y. Cho, J. Jang, S. Longpre, H. Lee, S. Yun, S. Shin, S. Kim, J. Thorne *et al.*, “Prometheus: Inducing fine-grained evaluation capability in language models,” in *The Twelfth International Conference on Learning Representations*, 2023.
- [19]
K. Ethayarajh, Y. Choi, and S. Swayamdipta, “Understanding dataset difficulty with v-usable information,” in *International Conference on Machine Learning*.   PMLR, 2022, pp. 5988–6008.
- [20]
N. Lambert, V. Pyatkin, J. Morrison, L. Miranda, B. Y. Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y. Choi *et al.*, “Rewardbench: Evaluating reward models for language modeling,” *arXiv preprint arXiv:2403.13787*, 2024.
- [21]
E. Frick, T. Li, C. Chen, W.-L. Chiang, A. N. Angelopoulos, J. Jiao, B. Zhu, J. E. Gonzalez, and I. Stoica, “How to evaluate reward models for rlhf,” *arXiv preprint arXiv:2410.14872*, 2024.
- [22]
T. Wang, D. Gui, Y. Hu, S. Lin, and L. Zhang, “Mpo: An efficient post-processing framework for mixing diverse preference alignment,” *arXiv preprint arXiv:2502.18699*, 2025.
- [23]
Z. Sun, Y. Shen, H. Zhang, Q. Zhou, Z. Chen, D. Cox, Y. Yang, and C. Gan, “Salmon: Self-alignment with instructable reward models,” *arXiv preprint arXiv:2310.05910*, 2023.
- [24]
Z. Liu, P. Wang, R. Xu, S. Ma, C. Ruan, P. Li, Y. Liu, and Y. Wu, “Inference-time scaling for generalist reward modeling,” *arXiv preprint arXiv:2504.02495*, 2025.
- [25]
X. Chen, G. Li, Z. Wang, B. Jin, C. Qian, Y. Wang, H. Wang, Y. Zhang, D. Zhang, T. Zhang *et al.*, “Rm-r1: Reward modeling as reasoning,” *arXiv preprint arXiv:2505.02387*, 2025.
- [26]
H. Peng, Y. Qi, X. Wang, Z. Yao, B. Xu, L. Hou, and J. Li, “Agentic reward modeling: Integrating human preferences with verifiable correctness signals for reliable reward systems,” *arXiv preprint arXiv:2502.19328*, 2025.
- [27]
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” *arXiv preprint arXiv:1707.06347*, 2017.
- [28]
Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu *et al.*, “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,” *arXiv preprint arXiv:2402.03300*, 2024.
- [29]
N. Razin, Z. Wang, H. Strauss, S. Wei, J. D. Lee, and S. Arora, “What makes a reward model a good teacher? an optimization perspective,” *arXiv preprint arXiv:2503.15477*, 2025.
- [30]
S. Kim, D. Kang, T. Kwon, H. Chae, D. Lee, and J. Yeo, “Rethinking reward model evaluation through the lens of reward overoptimization,” *arXiv preprint arXiv:2505.12763*, 2025.
- [31]
J. Park, S. Jwa, M. Ren, D. Kim, and S. Choi, “Offsetbias: Leveraging debiased data for tuning evaluators,” *arXiv preprint arXiv:2407.06551*, 2024.
- [32]
J. Ji, D. Hong, B. Zhang, B. Chen, J. Dai, B. Zheng, T. Qiu, B. Li, and Y. Yang, “Pku-saferlhf: Towards multi-level safety alignment for llms with human preference,” *arXiv preprint arXiv:2406.15513*, 2024.
- [33]
P. Röttger, H. R. Kirk, B. Vidgen, G. Attanasio, F. Bianchi, and D. Hovy, “Xstest: A test suite for identifying exaggerated safety behaviours in large language models,” *arXiv preprint arXiv:2308.01263*, 2023.
- [34]
J. Eisenstein, C. Nagpal, A. Agarwal, A. Beirami, A. D’Amour, D. Dvijotham, A. Fisch, K. Heller, S. Pfohl, D. Ramachandran *et al.*, “Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking,” *arXiv preprint arXiv:2312.09244*, 2023.
- [35]
Z. Wang, C. Nagpal, J. Berant, J. Eisenstein, A. D’Amour, S. Koyejo, and V. Veitch, “Transforming and combining rewards for aligning large language models,” *arXiv preprint arXiv:2402.00742*, 2024.
- [36]
W. Shi and X. Jin, “Heimdall: test-time scaling on the generative verification,” *arXiv preprint arXiv:2504.10337*, 2025.
- [37]
N. Singhi, H. Bansal, A. Hosseini, A. Grover, K.-W. Chang, M. Rohrbach, and A. Rohrbach, “When to solve, when to verify: Compute-optimal problem solving and generative verification for llm reasoning,” *arXiv preprint arXiv:2504.01005*, 2025.
- [38]
Y. Weng, M. Zhu, F. Xia, B. Li, S. He, S. Liu, B. Sun, K. Liu, and J. Zhao, “Large language models are better reasoners with self-verification,” *arXiv preprint arXiv:2212.09561*, 2022.
- [39]
J. Zhao, R. Liu, K. Zhang, Z. Zhou, J. Gao, D. Li, J. Lyu, Z. Qian, B. Qi, X. Li *et al.*, “Genprm: Scaling test-time compute of process reward models via generative reasoning,” *arXiv preprint arXiv:2504.00891*, 2025.
- [40]
K. De Langis, R. Koo, and D. Kang, “Dynamic multi-reward weighting for multi-style controllable generation,” *arXiv preprint arXiv:2402.14146*, 2024.
- [41]
A. Huang, A. Block, Q. Liu, N. Jiang, D. J. Foster, and A. Krishnamurthy, “Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment,” *arXiv preprint arXiv:2503.21878*, 2025.
- [42]
R. Liu, F. Bai, Y. Du, and Y. Yang, “Meta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning,” *Advances in Neural Information Processing Systems*, vol. 35, pp. 22 270–22 284, 2022.
- [43]
Z. Yu, W. Gu, Y. Wang, Z. Zeng, J. Wang, W. Ye, and S. Zhang, “Outcome-refining process supervision for code generation,” *arXiv preprint arXiv:2412.15118*, 2024.
- [44]
Q. Xie, Q. Li, Z. Yu, Y. Zhang, Y. Zhang, and L. Yang, “An empirical analysis of uncertainty in large language model evaluations,” *arXiv preprint arXiv:2502.10709*, 2025.
- [45]
K. Lu, H. Yuan, R. Lin, J. Lin, Z. Yuan, C. Zhou, and J. Zhou, “Routing to the expert: Efficient reward-guided ensemble of large language models,” *arXiv preprint arXiv:2311.08692*, 2023.
- [46]
J. Fu, X. Zhao, C. Yao, H. Wang, Q. Han, and Y. Xiao, “Reward shaping to mitigate reward hacking in rlhf,” *arXiv preprint arXiv:2502.18770*, 2025.
- [47]
T. Coste, U. Anwar, R. Kirk, and D. Krueger, “Reward model ensembles help mitigate overoptimization,” *arXiv preprint arXiv:2310.02743*, 2023.
- [48]
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell *et al.*, “Language models are few-shot learners,” *Advances in neural information processing systems*, vol. 33, pp. 1877–1901, 2020.
- [49]
R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen *et al.*, “Palm 2 technical report,” *arXiv preprint arXiv:2305.10403*, 2023.
- [50]
OpenAI, “Gpt-4 technical report,” 2023.
- [51]
Y. Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung *et al.*, “A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity,” *arXiv preprint arXiv:2302.04023*, 2023.
- [52]
G. E. Hinton, S. Osindero, and Y. W. Teh, “A fast learning algorithm for deep belief nets,” *Neural Computation*, vol. 18, pp. 1527–1554, 2006.
- [53]
I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, *Deep learning*.   MIT Press, 2016, vol. 1.
- [54]
Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-instruct: Aligning language model with self generated instructions,” *arXiv preprint arXiv:2212.10560*, 2022.
- [55]
J. Dodge, G. Ilharco, R. Schwartz, A. Farhadi, H. Hajishirzi, and N. Smith, “Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping,” *arXiv preprint arXiv:2002.06305*, 2020.
- [56]
C. Sun, X. Qiu, Y. Xu, and X. Huang, “How to fine-tune bert for text classification?” in *Chinese Computational Linguistics: 18th China National Conference, CCL 2019, Kunming, China, October 18–20, 2019, Proceedings 18*.   Springer, 2019, pp. 194–206.
- [57]
L. Hirschman and R. Gaizauskas, “Natural language question answering: the view from here,” *natural language engineering*, vol. 7, no. 4, pp. 275–300, 2001.
- [58]
T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee *et al.*, “Natural questions: a benchmark for question answering research,” *Transactions of the Association for Computational Linguistics*, vol. 7, pp. 453–466, 2019.
- [59]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” *Advances in neural information processing systems*, vol. 30, 2017.
- [60]
Y. Chang, X. Wang, J. Wang, Y. Wu, K. Zhu, H. Chen, L. Yang, X. Yi, C. Wang, Y. Wang *et al.*, “A survey on evaluation of large language models,” *arXiv preprint arXiv:2307.03109*, 2023.
- [61]
L. Tunstall, L. Von Werra, and T. Wolf, *Natural language processing with transformers*.   " O’Reilly Media, Inc.", 2022.
- [62]
V. Godbole, G. E. Dahl, J. Gilmer, C. J. Shallue, and Z. Nado, “Deep learning tuning playbook,” 2023, version 1.0. [Online]. Available: [http://github.com/google-research/tuning_playbook](http://github.com/google-research/tuning_playbook)
- [63]
A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Glue: A multi-task benchmark and analysis platform for natural language understanding,” in *International Conference on Learning Representations*, 2018.
- [64]
L. Yang, S. Zhang, L. Qin, Y. Li, Y. Wang, H. Liu, J. Wang, X. Xie, and Y. Zhang, “Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective,” *arXiv preprint arXiv:2211.08073*, 2022.
- [65]
A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan *et al.*, “The llama 3 herd of models,” *arXiv preprint arXiv:2407.21783*, 2024.
- [66]
E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen *et al.*, “Lora: Low-rank adaptation of large language models,” in *International Conference on Learning Representations*, 2021.
- [67]
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano *et al.*, “Training verifiers to solve math word problems,” *arXiv preprint arXiv:2110.14168*, 2021.
- [68]
O. Sainz, J. A. Campos, I. García-Ferrero, J. Etxaniz, O. L. de Lacalle, and E. Agirre, “Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark,” *arXiv preprint arXiv:2310.18018*, 2023.
- [69]
J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He, “Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters,” in *Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining*, 2020, pp. 3505–3506.
- [70]
S. Rajbhandari, O. Ruwase, J. Rasley, S. Smith, and Y. He, “Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning,” in *Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis*, 2021, pp. 1–14.
- [71]
L. Yang, S. Zhang, Z. Yu, G. Bao, Y. Wang, J. Wang, R. Xu, W. Ye, X. Xie, W. Chen *et al.*, “Supervised knowledge makes large language models better in-context learners,” *arXiv preprint arXiv:2312.15918*, 2023.
- [72]
C. Wang, X. Liu, Y. Yue, X. Tang, T. Zhang, C. Jiayang, Y. Yao, W. Gao, X. Hu, Z. Qi *et al.*, “Survey on factuality in large language models: Knowledge, retrieval and domain-specificity,” *arXiv preprint arXiv:2310.07521*, 2023.
- [73]
T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models are zero-shot reasoners,” *Advances in neural information processing systems*, vol. 35, pp. 22 199–22 213, 2022.
- [74]
D. Jiang, X. Ren, and B. Y. Lin, “Llm-blender: Ensembling large language models with pairwise ranking and generative fusion,” *arXiv preprint arXiv:2306.02561*, 2023.
- [75]
S. Kim, J. Suk, J. Y. Cho, S. Longpre, C. Kim, D. Yoon, G. Son, Y. Cho, S. Shafayat, J. Baek *et al.*, “The biggen bench: A principled benchmark for fine-grained evaluation of language models with language models,” *arXiv preprint arXiv:2406.05761*, 2024.
- [76]
G. Cui, L. Yuan, N. Ding, G. Yao, B. He, W. Zhu, Y. Ni, G. Xie, R. Xie, Y. Lin *et al.*, “Ultrafeedback: Boosting language models with scaled ai feedback,” *arXiv preprint arXiv:2310.01377*, 2023.
- [77]
A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu, “Qwen2.5 technical report,” *arXiv preprint arXiv:2412.15115*, 2024.
- [78]
A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu, W. Ge, X. Deng, X. Zhou, X. Ren, X. Zhang, X. Wei, X. Ren, Y. Fan, Y. Yao, Y. Zhang, Y. Wan, Y. Chu, Y. Liu, Z. Cui, Z. Zhang, and Z. Fan, “Qwen2 technical report,” *arXiv preprint arXiv:2407.10671*, 2024.
- [79]
Q. Team, “Qwen3,” April 2025. [Online]. Available: [https://qwenlm.github.io/blog/qwen3/](https://qwenlm.github.io/blog/qwen3/)
- [80]
G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu, “Hybridflow: A flexible and efficient rlhf framework,” *arXiv preprint arXiv: 2409.19256*, 2024.
- [81]
C. Zhang, G. Sheng, S. Liu, J. Li, Z. Feng, Z. Liu, X. Liu, X. Jia, Y. Peng, H. Lin *et al.*, “A framework for training large language models for code generation via proximal policy optimization,” in *NL2Code Workshop of ACM KDD*, 2024.
- [82]
G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière *et al.*, “Gemma 3 technical report,” *arXiv preprint arXiv:2503.19786*, 2025.
- [83]
G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican *et al.*, “Gemini: a family of highly capable multimodal models,” *arXiv preprint arXiv:2312.11805*, 2023.
- [84]
D. Mahan, D. Van Phung, R. Rafailov, C. Blagden, N. Lile, L. Castricato, J.-P. Fränken, C. Finn, and A. Albalak, “Generative reward models,” *arXiv preprint arXiv:2410.12832*, 2024.
- [85]
S. Pitis, Z. Xiao, N. Le Roux, and A. Sordoni, “Improving context-aware preference modeling for language models,” *Advances in Neural Information Processing Systems*, vol. 37, pp. 70 793–70 827, 2024.
- [86]
J. Ruan, W. Wang, and X. Wan, “Defining and detecting vulnerability in human evaluation guidelines: A preliminary study towards reliable nlg evaluation,” *arXiv preprint arXiv:2406.07935*, 2024.
- [87]
C. Van Der Lee, A. Gatt, E. Van Miltenburg, S. Wubben, and E. Krahmer, “Best practices for the human evaluation of automatically generated text,” in *Proceedings of the 12th International Conference on Natural Language Generation*, 2019, pp. 355–368.
- [88]
A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan *et al.*, “Deepseek-v3 technical report,” *arXiv preprint arXiv:2412.19437*, 2024.
- [89]
Z. Li, X. Xu, T. Shen, C. Xu, J.-C. Gu, Y. Lai, C. Tao, and S. Ma, “Leveraging large language models for nlg evaluation: Advances and challenges,” *arXiv preprint arXiv:2401.07103*, 2024.
- [90]
Z. Yu, C. Gao, W. Yao, Y. Wang, Z. Zeng, W. Ye, J. Wang, Y. Zhang, and S. Zhang, “Freeeval: A modular framework for trustworthy and efficient evaluation of large language models,” *arXiv preprint arXiv:2404.06003*, 2024.
- [91]
D. Li, B. Jiang, L. Huang, A. Beigi, C. Zhao, Z. Tan, A. Bhattacharjee, Y. Jiang, C. Chen, T. Wu *et al.*, “From generation to judgment: Opportunities and challenges of llm-as-a-judge,” *arXiv preprint arXiv:2411.16594*, 2024.
- [92]
J. Ying, Y. Cao, Y. Bai, Q. Sun, B. Wang, W. Tang, Z. Ding, Y. Yang, X. Huang, and S. Yan, “Automating dataset updates towards reliable and timely evaluation of large language models,” *arXiv preprint arXiv:2402.11894*, 2024.
- [93]
M. Ravaut, B. Ding, F. Jiao, H. Chen, X. Li, R. Zhao, C. Qin, C. Xiong, and S. Joty, “How much are large language models contaminated? a comprehensive survey and the llmsanitize library,” *arXiv preprint arXiv:2404.00699*, 2024.
- [94]
E. Frick, C. Chen, J. Tennyson, T. Li, W.-L. Chiang, A. N. Angelopoulos, and I. Stoica, “Prompt-to-leaderboard,” *arXiv preprint arXiv:2502.14855*, 2025.
- [95]
L. Gao, J. Schulman, and J. Hilton, “Scaling laws for reward model overoptimization,” in *International Conference on Machine Learning*.   PMLR, 2023, pp. 10 835–10 866.
- [96]
Y. Yu, Z. Chen, A. Zhang, L. Tan, C. Zhu, R. Y. Pang, Y. Qian, X. Wang, S. Gururangan, C. Zhang *et al.*, “Self-generated critiques boost reward modeling for language models,” *arXiv preprint arXiv:2411.16646*, 2024.
- [97]
Z. Ankner, M. Paul, B. Cui, J. D. Chang, and P. Ammanabrolu, “Critique-out-loud reward models,” *arXiv preprint arXiv:2408.11791*, 2024.
- [98]
Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon *et al.*, “Constitutional ai: Harmlessness from ai feedback,” *arXiv preprint arXiv:2212.08073*, 2022.
- [99]
S. Tang, Y. Wang, and C. Jin, “Is elo rating reliable? a study under model misspecification,” *arXiv preprint arXiv:2502.10985*, 2025.
- [100]
J. Lindsey, W. Gurnee, E. Ameisen, B. Chen, A. Pearce, N. L. Turner, C. Citro, D. Abrahams, S. Carter, B. Hosmer, J. Marcus, M. Sklar, A. Templeton, T. Bricken, C. McDougall, H. Cunningham, T. Henighan, A. Jermyn, A. Jones, A. Persic, Z. Qi, T. B. Thompson, S. Zimmerman, K. Rivoire, T. Conerly, C. Olah, and J. Batson, “On the biology of a large language model,” *Transformer Circuits Thread*, 2025. [Online]. Available: [https://transformer-circuits.pub/2025/attribution-graphs/biology.html](https://transformer-circuits.pub/2025/attribution-graphs/biology.html)
- [101]
OpenAI, “Gpt-4.1 and gpt-4.1 nano overview,” [https://openai.com/blog](https://openai.com/blog), 2024, accessed: 2025-05-15.
- [102]
K. Kavukcuoglu, “Gemini 2.5: Our most intelligent ai model,” [https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/](https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/), 2025, accessed: 2025-05-15.
- [103]
A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei *et al.*, “Qwen2. 5 technical report,” *arXiv preprint arXiv:2412.15115*, 2024.
- [104]
G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière *et al.*, “Gemma 3 technical report,” *arXiv preprint arXiv:2503.19786*, 2025.
- [105]
W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in *Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles*, 2023.

## Appendix A What Makes A Good Principle? Analyzing Biases and Principle Quality

The advent of principle-following reward models like RewardAnything shifts a significant part of the alignment control to the user-provided natural language principle. This also introduces a new consideration: the quality and formulation of the principle itself. To investigate this, we conducted experiments using RewardAnything on the RM-Bench "hard" split. This split is particularly challenging as it often presents scenarios where responses that are incorrect but detailed and well-formatted are pitted against responses that are correct but concise. Length and Correctness should have been two orthogonal aspects of text quality, and RM-Bench carefully curated comparisons between responses with varying degrees of these two aspects, to judge whether reward models could consistently choose the correct responses regardless of the length aspect.

However, traditional reward models can struggle here, sometimes favoring length or superficial cues over factual accuracy. Our goal was to understand how varying the *priority* of objectives and the *clarity* of the principle influences RewardAnything’s ability to navigate such trade-offs and adhere to the intended evaluation logic. The findings, visually summarized in [Figure 3](https://arxiv.org/html/2506.03637v2#S6.F3) in the main text, offer crucial insights for users aiming to craft effective principles.

**Table 5: Principles for Priority and Clarity Experiments on RM-Bench Hard Split.**
| Experiment | Level | Principle Content |
| --- | --- | --- |
| Priority | Length Only | I want only long responses; accuracy and other factors should be completely ignored. |
| Priority | Length > Accuracy | I want long responses; only consider correctness if two responses are the same or very similar in length, otherwise judge primarily according to length. |
| Priority | Length = Accuracy | I want responses that are both correct and long; these should be treated equally. Responses that are long and correct should be best. Either missing length or correctness makes it less good. Worst responses are short and concise. |
| Priority | Length < Accuracy | Focus more on correctness; consider length only if correctness is equal. |
| Priority | Accuracy Only | Focus solely on correctness; length and other factors should be completely ignored. |
| Clarity | Vague | Good responses are preferred. |
| Clarity | Objectives Only | Good responses should be accurate and detailed. |
| Clarity | Objectives + Structured Rules | Correctness is paramount for good responses. Between two correct responses, the more detailed one is better. Finally, if correctness and detail are equal, a well-formatted response is preferred. Crucially, a correct and concise response is always better than an incorrect and detailed one. |

### A.1 The Importance of Goal Prioritization

Many real-world alignment tasks involve multiple, sometimes conflicting, objectives. We explored how explicitly stating different priorities for different goals impacts performance.

The "Priority" plot in [Figure 3](https://arxiv.org/html/2506.03637v2#S6.F3) illustrates this. When the principle provided to RewardAnything gave no clear priority, such as "Length = Accuracy" (specifying equal importance for these two aspects), the model’s performance on the "hard" instances was notably lower. In this scenario, RewardAnything behaves more like traditional discriminative RMs, which, lacking explicit guidance on how to weigh conflicting signals like correctness versus verbosity, may struggle to consistently identify the truly preferred (i.e., correct) response. This suggests that when faced with underspecified priorities, a principle-following RM might still grapple with ambiguities in a way that mirrors the challenges of learning from implicit preferences.

Conversely, when the principle explicitly prioritized accuracy over length (e.g., "Accuracy Only" or "Length < Accuracy"), RewardAnything’s performance on the "hard" split improved dramatically. This aligns perfectly with the design goal of the RM-Bench "hard" split, which is to reward models that can discern factual correctness even when it’s presented less elaborately. This finding underscores a critical lesson: *users must first have a clear understanding of their own objectives and, crucially, the relative importance or priority among these objectives, especially when they might conflict.*

Conclusion on Priority: For principle-following RMs to be effective, especially in complex scenarios with multiple objectives, it is paramount that the user clearly defines the hierarchy or priority of these goals within the principle itself. Ambiguous or unstated priorities can lead the RM to suboptimal performance, potentially mimicking the biases or confusion observed in traditional models. Explicit prioritization empowers the RM to resolve conflicts in line with the user’s true intent.

### A.2 The Role of Clarity and Structure in Multi-Objective Principles

Beyond prioritizing goals, the overall clarity and structure of the principle, particularly when multiple objectives are involved, play a significant role. The "Clarity" plot in [Figure 3](https://arxiv.org/html/2506.03637v2#S6.F3) explores this by comparing principles that are "Vague," those that list "Objectives Only," and those that provide "Objectives + Structured Rules."

An interesting, perhaps counter-intuitive, finding is that providing "Objectives Only" (e.g., "The response should be accurate and concise") can sometimes lead to *worse* performance than a "Vague" principle, especially on the "hard" split. One interpretation is that merely listing multiple objectives without guidance on their interplay or application can introduce ambiguity or even internal conflict for the model. It essentially hands the complex task of interpreting and balancing these potentially competing goals back to the RM, which might then struggle more than if it were operating under a single, albeit general, directive.

The most significant performance gains were observed when the principle included not just the objectives but also structured rules or heuristics on how to apply them. For example, instead of just saying "be accurate and concise," a structured principle might add, "If a response is inaccurate, it is undesirable regardless of its conciseness. Among accurate responses, prefer the more concise one." This kind of structured guidance helps the model navigate the decision-making process more effectively.

Conclusion on Clarity: When crafting principles with multiple objectives, simply enumerating the desired qualities is often insufficient and can even be detrimental. It is far more effective to provide *clear, structured guidance* on how these objectives should be interpreted, applied, and potentially prioritized relative to one another. Providing such "rules of engagement" reduces ambiguity, constrains the model’s interpretation in a desirable way, and leads to more robust and predictable adherence to the user’s intended evaluation logic.

Furthermore, a crucial implication of these findings is that the "biases" often observed in traditional benchmarks or exhibited by standard reward models can be significantly mitigated, if not largely eliminated, when RewardAnything is guided by a well-formulated principle. By providing a clear principle such as "Accuracy Only" or "Length < Accuracy"—which explicitly encodes the benchmark’s intended preference—RewardAnything’s performance surged to 82.5 and 81.6, respectively. This demonstrates that RewardAnything, equipped with an appropriate principle, can effectively override superficial or misleading correlations (like length preference) and align its judgments much more closely with the specific, intended criteria of a task or benchmark. This capability is vital for achieving more robust and genuinely meaningful evaluations and alignments, moving beyond simple pattern matching to a more nuanced understanding of specified human values.

## Appendix B Comparing with Pointwise and Pairwise Reward Models

We analyze the limitations of two dominant reward modeling paradigms: pointwise scoring and pairwise comparison, and highlight the advantages of our principle-following approach grounded in natural language instructions.

#### Pointwise Scoring.

This method assigns an independent quality score to each response, either by prompting a general purpose LLM, or training a sequence-classifier on regression-like tasks. It is simple, computationally efficient, and well-suited for large-scale pretraining. A notable advantage is its high reward variance, which offers strong learning signals during reinforcement learning. As shown in [Table 2](https://arxiv.org/html/2506.03637v2#S5.T2) and supported by Razin et al. [29], higher variance rewards helps better for distinguishing better outputs from all responses. However, pointwise scoring overlooks relative differences between responses, often leading to shallow preference understanding and weaker generalization.

**Table 6: Comparison of computational costs across reward modeling paradigms.**
| Method | LLM Calls | LLM Tokens |
| --- | --- | --- |
| Pointwise Scoring | $\Theta(n)$ | $\Theta(n)$ |
| Pairwise Comparison | $\Theta(n^{2})$ | $\Theta(n^{2})$ |
| Listwise Scoring (Ours) | $\Theta(1)$ | $\Theta(n)$ |

#### Pairwise Comparison.

This paradigm compares two responses to determine preference, aligning better with human judgment and widely adopted in RLHF. Yet, it incurs quadratic inference costs, requiring $\binom{n}{2}$ comparisons to rank $n$ responses—limiting deployment efficiency in recent RL algorithms. In practice, for instance, GRPO often requires generating N = 10 rollout responses for each one prompt, and this would result in 45 pairwise comparison calls in worst case. As some of these pairwise comparison methods (e.g. RM-R1 [25] and [24] which are very recent concurrent works), each pairwise comparison call requires inference-time best-of-n sampling along with reasoning steps which requires generating long chain-of-thought sequences, such approaches become computationally infeasible as the reward process would simply take too long and too expensive. In contrast, our model scores and ranks all candidates with a single LLM inference call, ensuring computational cost.

More importantly, pairwise training typically uses implicit "chosen vs. rejected" labels without clear criteria. As noted by Liu et al. [24], such supervision lacks semantic clarity, making it difficult to align with human intent. Our method leverages explicit natural language principles, allowing the model to reason about preference criteria in a transparent and controllable way.

While some work transforms pairwise outcomes into Elo scores, these are often unstable under non-transitive or inconsistent preferences [99], limiting reliability in complex scenarios.

#### Conclusion.

Pointwise and pairwise approaches offer practical benefits but fall short in expressiveness, efficiency, and interpretability. Our principle-following method bridges these gaps by grounding reward models in natural language, enabling more flexible, interpretable, and scalable alignment with human values.

## Appendix C Experiment Setup: Dataset Details

RewardBench[20] is a benchmark dataset and codebase designed to systematically evaluate reward models (RMs) across diverse and challenging scenarios. It addresses a critical gap in the open-source RLHF ecosystem, where resources for training and understanding reward models remain limited. The dataset comprises structured prompt–chosen–rejected triplets covering domains such as chat, reasoning, and safety, and includes carefully constructed comparison cases with subtle but verifiable errors—such as factual inaccuracies or logical flaws—that justify a clear preference. This design allows for the fine-grained evaluation of how well reward models align with human values under distributional shifts and nuanced judgments. RewardBench supports benchmarking of models trained using various strategies, including standard supervised (MLE-based) classifiers and more advanced methods like Direct Preference Optimization (DPO). In addition to ranking accuracy, the benchmark also reveals systematic model behaviors such as over-refusal tendencies, limitations in reasoning, and difficulties with instruction-following—thus contributing to a more transparent and rigorous understanding of the reward modeling process within RLHF.

RM-BENCH[7] is a benchmark specifically designed to evaluate the capability of reward models to distinguish fine-grained differences in content and to resist superficial style biases. Unlike prior benchmarks that often conflate model quality with model size—by comparing responses generated by weaker vs. stronger LMs—RM-BENCH ensures that both the preferred and dispreferred responses are produced by the same language model (e.g., GPT-4o), with subtle modifications introduced to the latter. This design isolates the content-sensitivity aspect and avoids style-related confounding factors. Furthermore, RM-BENCH introduces controlled stylistic variations (e.g., concise vs. markdown-formatted responses) to test robustness against stylistic distractions. Its performance metric correlates highly with post-RLHF policy model quality, making it a reliable proxy for reward model effectiveness. Evaluations on nearly 40 reward models show that even state-of-the-art models achieve modest accuracy (e.g.,  46.6% under style interference), highlighting significant room for improvement in current reward model alignment.

XSTest[33] is a safety-focused benchmark developed to identify exaggerated safety behaviors in large language models—that is, cases where models refuse safe prompts simply because they resemble unsafe ones. The dataset consists of 250 carefully crafted safe prompts across 10 categories that a well-calibrated model should answer, along with 200 matched unsafe prompts that should appropriately be rejected. This contrastive setup helps probe whether models can balance helpfulness and harmlessness. XSTest reveals common failure modes in modern LLMs: some models refuse legitimate queries due to lexical similarity to dangerous inputs, reflecting overfitted safety filters. The benchmark is particularly useful in evaluating how system prompts and alignment strategies influence a model’s refusal behavior. Empirical results show varied safety-performance trade-offs across models such as GPT-4, Llama2, and Mistral. GPT-4 strikes the best balance, while others either over-refuse safe prompts or under-refuse unsafe ones. XSTest thus serves as a valuable tool for understanding and mitigating the tension between safety and usability in LLMs.

## Appendix D Experiment Setup: Models

This section provides brief descriptions of the major language models used throughout our experiments. These models span a diverse range of architectures, sizes, and capabilities, including both proprietary and open-weight models.

GPT-4.1[101] is OpenAI’s latest flagship model, demonstrating strong and stable performance across natural language understanding, reasoning, and multi-turn dialogue tasks.

GPT-4.1 Nano[101] is a lightweight variant of the GPT-4.1 series, optimized for cost and speed, offering significantly lower inference latency while maintaining a solid level of capability, making it suitable for edge deployments or cost-sensitive scenarios.

Claude 3.5 Haiku[100] is Anthropic’s fastest model to date, featuring low latency and significantly improved instruction-following and coding capabilities. It outperforms even Claude 3.5 Opus on several intelligence benchmarks.

Gemini 2.5 Pro[102] is Google’s latest state-of-the-art, natively multimodal model based on a Mixture-of-Experts architecture, capable of handling complex reasoning across text, images, audio, video, and code, with support for million-token context windows.

DeepSeek V3[88] is a high-performance Mixture-of-Experts (MoE) model employing innovative Multi-head Latent Attention (MLA) and a novel loss-free balancing strategy. Pretrained on 14.8 trillion tokens, it achieves strong performance and stable training with minimal compute.

Qwen2.5-1.5B / 7B / 72B[103] The Qwen2.5 series, developed by Alibaba, includes models of various sizes from lightweight to large-scale. With 18 trillion tokens for pretraining and multi-stage fine-tuning, they excel in language understanding, mathematics, code generation, and instruction-following.

Gemma3-1B / 12B[104] Gemma 3 is Google’s lightweight multimodal model series with strong multilingual and long-context capabilities. Its architectural changes reduce KV-cache overhead, and its improved post-training leads to superior performance over the previous Gemma 2 generation.

LLaMA 3.1 8B[65] is part of Meta’s latest open-source release, supporting 128K context length and enhanced capabilities in reasoning, multilingual translation, and tool use. It is well-suited for long-form summarization, chat agents, and coding assistants.

## Appendix E Experiment Setup: Hyperparameters And Training Details

We release hyperparameters, training details here. We use verl [81] along with EasyR1 [80] to train our models, and use the following hyperparameters. All our models were trained on a NVIDIA-A100-80G-SXM4 cluster, but for inference, consumer-grade GPUs like NVIDIA RTX 3090, 4090 and 5090 works just fine. For inference, we use vLLM [105] and limit the output to 2048 tokens.

**Table 7: Key hyperparameters used for training.**
| Hyperparameter | Value |
| --- | --- |
| Data Configuration |  |
| Max Prompt Length | 2048 |
| Max Response Length | 2048 |
| Rollout Batch Size | 768 |
| Validation Batch Size | 1536 |
| Seed | 1 |
| Algorithm Configuration |  |
| Adversarial Estimator | grpo |
| KL Coefficient ($\beta_{\text{KL}}$) | $1.0\times 10^{-2}$ |
| Worker: Actor Configuration |  |
| Global Batch Size | 192 |
| Micro Batch Size (Update, per device) | 4 |
| Micro Batch Size (Experience, per device) | 16 |
| Max Gradient Norm | 1.0 |
| Learning Rate (LR) | $1.0\times 10^{-6}$ |
| Weight Decay | $1.0\times 10^{-2}$ |
| Optimizer Strategy | AdamW |
| LR Warmup Ratio | 0.0 |
| Worker: Rollout Configuration |  |
| Number of Rollout Trajectories (n) | 20 |
| Temperature | 1.0 |
| Top-p | 0.99 |
| Trainer Configuration |  |
| Total Epochs | 10 |
| Number of Nodes | 3 |
| GPUs per Node | 8 |
| Validation Frequency (epochs) | 1 |
| Save Frequency (epochs) | 1 |

## Appendix F Concensus Algorithm for aggregating rankings from different LLM judges

[Figure 5](https://arxiv.org/html/2506.03637v2#A6.F5) contains the pseudo code for our concensus algorithm which is used to find the longest subsequence that at least K of all judge LLMs would agree on. This is similar to majority voting in terms of merging decisions of different evaluators, but works on multiple sequences containing ranked results.

Figure: Algorithm 1 Consensus Ranking Generation

## Appendix G Prompts Used and Examples

We include the complete system prompt used for our model and all generative reward models, for all the experiments. We applied the exact same system prompt for all models unless explicitly stated. The principles, prompts and responses are fed into the models as user prompts. [Figure 6](https://arxiv.org/html/2506.03637v2#A9.F6) shows our system prompt. [Figure 7](https://arxiv.org/html/2506.03637v2#A9.F7) and [Figure 8](https://arxiv.org/html/2506.03637v2#A9.F8) present an input-output example from RewardAnything.

## Appendix H Use of AI Assistants

Large language models are used solely for grammar correction, wording refinement, and formatting adjustments in the preparation of this paper. No scientific content or research findings are generated by AI. All content is written by the authors and is thoroughly reviewed and verified for accuracy and integrity. The use of AI tools does not affect the originality of the work or the authors’ responsibility for its content.

## Appendix I Limitations

Our work introduces RewardAnything, a novel reward model paradigm that follows natural language principles. While this approach significantly lowers the barrier for conducting diverse alignment studies, it also brings to the forefront the challenge of ensuring the robustness and safety of the principles themselves. Our analysis in  [Appendix A](https://arxiv.org/html/2506.03637v2#A1) provides initial insights and recommendations on crafting effective principles. However, a comprehensive exploration of all facets—such as the sensitivity of RewardAnything to subtle variations in principle phrasing, the potential for adversarial manipulations of principles, or the difficulty in exhaustively predicting all downstream behavioral consequences of a given principle—is beyond the scope of this work. Much like the extensive field of prompt engineering for instruction-following models, the rigorous study of how to design, validate, and ensure the safety of principles for RMs like RewardAnything presents a rich set of open questions for future research.

Figure: Figure 6: The system prompt for RewardAnything . The full input prompt for RewardAnything is a concatenation of the system prompt and the user prompt. We release a clean implementation in our Python package: [https://pypi.org/project/rewardanything/](https://pypi.org/project/rewardanything/)

Figure: Figure 7: An example input of RewardAnything . This is the formatted version of the user prompt, and the actual input prompt is organized with JSON format. The full input prompt for RewardAnything is a concatenation of the system prompt and the user prompt.

Figure: Figure 8: An example output of RewardAnything . We release a clean implementation in our Python package that takes care of the output parsing: [https://pypi.org/project/rewardanything/](https://pypi.org/project/rewardanything/)

Figure: Figure 9: A simple example of how to use our rewardanything Python package. The API is designed to be intuitive, allowing for easy integration and use. For more details and documentation, please visit our project page: [%****␣appendix.tex␣Line␣425␣****https://zhuohaoyu.github.io/RewardAnything/](%****%20appendix.tex%20Line%20425%20****https://zhuohaoyu.github.io/RewardAnything/)

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. Are there any existing reward models trained via reinforcement learning (RL) that the authors compare their approach against? It would be helpful to understand how the proposed method performs relative to prior RL-based reward models in terms of generalization, adaptability, and bias control.

2. Could you report the REWARDANYTHING on REWARD bench?

3. How does instruction following capability correlates with the principle adherence capability, if we start with a IF tuned model (verifiy-if) would that results in even better performance? 

4. How could we combine this with rubrics based rewards ideas? The model seemed to follow a set of principles which are analogous to rubrics (checks) which gains popularity recently for evaluating the response quality.
soundness: 2
strengths: The paper presents a clear and coherent narrative — progressing logically from the motivation of improving reward model generalization to the development of RABENCH and the proposed principle-following reward model. The methodology is well-structured and easy to follow. The authors further strengthen their contribution by conducting on-policy training with GRPO, featuring an intuitive reward design. Experimental results demonstrate promising generalization of the learned reward model when applied to diverse downstream tasks.
summary: The paper identifies two major flaws in current reward models: (1) limited generalization and adaptability to new tasks, and (2) the lack of explicit bias control and interpretability due to reliance on implicit preference learning. To address these limitations, the authors propose a principle-following reward model that extends beyond fixed reward criteria. Analogous to instruction-following large language models, this framework enables the reward model to interpret and generalize over arbitrary principles (P) — which may represent task specifications, ethical guidelines, or combinations of multiple criteria with defined priorities. The model is expected to handle unseen principles at varying levels of specificity and abstraction.

To systematically evaluate this capability, the authors introduce RABENCH, a comprehensive benchmark designed to test reward models on their ability to adapt evaluation criteria according to explicitly provided natural-language principles.

Building on this, they develop Reward-Anything, a reward model trained via on-policy reinforcement learning. The objective is to encourage the model to produce correctly ranked lists consistent with the guiding principles, thereby promoting both adaptability and principled generalization.
weaknesses: Didn't report performance on other popular benchmark like RewardBench to show generalization, and some other popular work uses reward model for test time compute (verifier for best of N), reporting numbers on these would make the generalization statement more robust.

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 2
questions: In addition to the weakness section, I have the following questions:

How exactly were the discriminative RMs prompted with the principle in the RM-Bench evaluation? Given their architecture, what evidence do you have that they are capable of utilizing this input? How would REWARDANYTHING perform if it were not provided with an explicit principle tailored to the benchmark?

Q2. Regarding W2: Given that the training data relies solely on LLM consensus without human oversight, how do you ensure that REWARDANYTHING is learning generalizable principle-following capabilities rather than overfitting to the specific patterns and biases of the LLM judges (e.g., GPT-4.1) used for data generation?
soundness: 2
strengths: Timely and Important Problem: The paper tackles a highly relevant problem. The inflexibility and bias of current RMs are major bottlenecks in deploying aligned AI systems. 

Efficiency and Interpretability: The listwise ranking approach of REWARDANYTHING is computationally efficient compared to pairwise comparisons, requiring only a single inference call to rank multiple candidates. Furthermore, the model's generation of explicit reasoning steps enhances the interpretability of the reward signal.
summary: This paper addresses a limitation in traditional Reward Modeling for RLHF: the rigidity of RMs trained on static preference datasets. Current RMs learn implicit preferences and struggle to adapt to new criteria without costly retraining. The authors propose a paradigm shift towards "principle-following reward models," which can dynamically adjust their evaluation criteria based on natural language instructions
weaknesses: Unfair Comparison Methodology on RM-Bench: The claimed SOTA performance on RM-Bench (Section 6.1, Table 2) appears to rely on a flawed comparison methodology. The authors state they provided REWARDANYTHING with a clear principle prioritizing accuracy, and that "this principle is also passed to other RMs as system prompt" (L355-356). However, the majority of the baselines, particularly the Discriminative Reward Models (e.g., Skywork-Reward, FsfairX, Nemotron), are fine-tuned sequence classifiers or regression models; they are not architecturally designed to interpret system prompts or dynamically adjust their learned preferences based on inference-time instructions. By giving REWARDANYTHING explicit instructions on the evaluation criteria, while the baselines are forced to rely on their implicit learned preferences, the comparison is heavily biased. This undermines the SOTA claim.

Complexity and Stability of GRPO Training: The proposed training method (GRPL) uses GRPO, an online RL algorithm, with a complex, multi-component reward function (Equations 2-4) involving numerous hyperparameters. This is likely significantly more complex and potentially less stable than standard SFT or offline methods like DPO. While the ablation (Table 4) shows GRPO outperforms SFT, the paper lacks analysis of the training costs, stability, and hyperparameter sensitivity of GRPO in this context. The necessity of this complexity over potentially adapting DPO for principle-following is not established.

The Burden of "Principle Engineering": The analysis in Section 6.1 (Figure 3) highlights that performance drops significantly when principles are vague or when multiple objectives lack clear prioritization. This suggests that the success of the method heavily depends on the user's ability to craft high-quality, structured principles. This potentially shifts the challenge from "data collection" to "principle engineering," which presents its own difficulties, and may limit the claimed generalization if the model fails on ambiguous real-world principles.

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 3
questions: 1. How each component of the dataset, such as the principles and the chosen/rejected responses, is evaluated?
2. How the principles were curated or how they influenced the generation of preference data and the training process?
3. How does your work differ from
- Improving Context-Aware Preference Modeling for Language Models, NeurIPS 2024, https://openreview.net/forum?id=52r4XJYzjg
- Deliberative Alignment: Reasoning Enables Safer Language Models, OpenAI 2024, https://arxiv.org/abs/2412.16339?
soundness: 3
strengths: 1. The paper addresses an important challenge in reward modeling: enabling flexible alignment of language models through natural language principles rather than static preference datasets.
2. The paper is mostly well-written and organized. Key methods like GRPO and the listwise reward structure are clearly explained. The paper is technically competent, and the proposed method is well-implemented.
3. The proposed dataset and benchmark are open source, which could be useful tools for future research in this space.
summary: The paper introduces REWARDANYTHING, a reward modeling framework that enables language models to follow arbitrary, natural language-specified principles at inference time. It functions as a language model judge trained on preference data using Group Relative Preference Optimization (GRPO). To evaluate this capability, the authors propose RABENCH, a new benchmark designed to assess how well reward models generalize to diverse and previously unseen evaluation principles. Their results show that existing reward models struggle with such generalization, whereas REWARDANYTHING performs strongly, often matching or surpassing the performance of models like GPT-4.1 on both traditional benchmarks and tasks requiring principle generalization.
weaknesses: 1. The paper lacks in-depth analysis regarding whether the model’s outputs, when conditioned on natural language principles, truly align with those principles. Section 4.2 provides only a very high-level description of how the quality of the preference data is ensured, offering no details on how each component of the dataset, such as the principles and the chosen/rejected responses, is evaluated. There is also no information about the annotator recruitment process or qualification criteria.
2. The paper does not clearly explain how the principles were curated. Since the dataset is a critical component of this work, a more thorough analysis of its quality is necessary. I only found a vague statement on line 163: “We manually curated 200 distinct principles,” along with some generic analysis in Appendix A. However, there is no clear explanation of how the principles were curated. Did you write the principles yourself, or were they sourced from existing datasets?
3. The idea of using LLMs as generative reward models conditioned on natural language and producing structured reasoning has already been explored in prior work [1, 2, 4]. Among these, [1] appears to be the most similar to the proposed approach. Based on my reading, the primary methodological difference is the use of GRPO for training the reward model in this work, where as [1] used pairwise reward model loss. Although [1] is listed in the references, it is neither discussed nor even mentioned in the main text, which is a significant omission. Please correct me if I’m mistaken. Furthermore, similar methods are already widely used in alignment research. For example, [3] aligns models using what they term “specs,” which are effectively equivalent to the natural language principles used here. Likewise, [5] introduces “checklist-guided evaluation” for alignment, where the checklists are derived from in-situ user interactions and feedback. The distinctions claimed in this work, such as GRPO training or explicit principle-following, appear superficial without a direct comparison to these existing approaches.

---
Related Work

[1] Improving Context-Aware Preference Modeling for Language Models, NeurIPS 2024, https://openreview.net/forum?id=52r4XJYzjg

[2] CARMO: Dynamic Criteria Generation for Context-Aware Reward Modelling, ACL 2025, https://arxiv.org/abs/2410.21545

[3] Deliberative Alignment: Reasoning Enables Safer Language Models, OpenAI 2024, https://arxiv.org/abs/2412.16339

[4] Generative Reward Models, Arxiv 2024, https://arxiv.org/abs/2410.12832

[5] WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback, Arxiv 2024, https://arxiv.org/abs/2408.15549


Decision: Reject