Bradley-Terry and Multi-Objective Reward Modeling Are Complementary
Authors: Zhiwei Zhang
Hui Liu
Xiaomin Li
Zhenwei Dai
Jingying Zeng
Fali Wang
Minhua Lin
Ramraj Chandradevan
Linlin Wu
Zhen Li
Chen Luo
Zongyu Wu
Xianfeng Tang
Qi He
Suhang Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models; Reward Models
Primary area: foundation or frontier models, including LLMs

Abstract: Reward models trained on human preference data have demonstrated strong effectiveness in aligning Large Language Models (LLMs) with human intent under the framework of Reinforcement Learning from Human Feedback (RLHF). However, RLHF remains vulnerable to reward hacking, where the policy exploits imperfections in the reward function rather than genuinely learning the intended behavior. 
Although significant efforts have been made to mitigate reward hacking, they predominantly focus on and evaluate in-distribution scenarios, where the training and testing data for the reward model share the same distribution.
In this paper, we empirically show that state-of-the-art methods struggle in more challenging out-of-distribution (OOD) settings. We further demonstrate that incorporating fine-grained multi-attribute scores helps address this challenge. However, the limited availability of high-quality data often leads to weak performance of multi-objective reward functions, which can negatively impact overall performance and become the bottleneck. To address this issue, we propose a unified reward modeling framework that jointly trains Bradley-Terry (BT) single-objective and multi-objective regression-based reward functions using a shared embedding space. We theoretically establish a connection between the BT loss and the regression objective and highlight their complementary benefits. Specifically, the regression task enhances the single-objective reward function’s ability to mitigate reward hacking in challenging OOD settings, while BT-based training improves the scoring capability of the multi-objective reward function, enabling a 7B model to outperform a 70B baseline.
Extensive experimental results demonstrate that our framework significantly improves both the robustness and the scoring performance of reward models.

Published as a conference paper at ICLR 2026
BRADLEY-TERRY AND MULTI-OBIECTIVE REWARD MODELING ARE COMPLEMENTARY
Zhiwei Zhang' Hui
Hui Liu Xiaomin Li", Zhenwei Jingying Zeng', Fali Wang', Minhua Lin', Ram Chandradevan", Linlin Wu', Zhen Li", Chen Luo", Zongyu Wu', Xianfeng Tang', Qi He²
Suhang Wanglt
'The Pennsylvania State University 2 AAmazon HHarvard University *University of Utah (zb25349, s2w4 94 @épsu.edu
ABSTRACT
Reward models trained on human preference data have demonstrated strong effec- tiveness in aligning Large Language Models (LLLMs) with human intent under the framework of Reinforcement Learning from Human Feedback (RLHF). However, RLHF remains vulnerable to reward hacking, where the policy exploits imperfec- tions in the reward function rather than genuinely learning the intended behavior. Although significant efforts have been made to mitigate reward hacking, they predominantly focus on and evaluate in-distribution scenarios, where the training and testing data for reward model share the same distribution. In this paper, empirically show that state-of -f-thh--tttttttt methods struggle in more challenging -iitti-diittitttionn (OOD) settings. We further demonstrate that incorporating fine- grained multi autti--ttribute scores helps address this challenge. However, the limited availability of high-quality data often leads to weak performance of multi-objective reward functions, which can negatively impact overall performance and become the bottleneck. To address this issue, we propose unified reward modeling frame- work that jointly trains Bradley-Terry (BT) single-objective and multi-objective regression-bars based reward functions using a shared embedding space. We theoreti- cally establish a connection between the BT loss and the regression objective and highlight their complementary benefits. Specifically, the regression task enhances the single-objective reward function's ability to mitigate reward hacking chal- lenging OOD settings, while BT-based training improves the scoring capability of the multi-objective reward function, enabling a 7B model to outperform a 70B base- line. Extensive experimental results demonstrate that our framework significantly improves both the robustness and the scoring performance of reward models.
INTRODUCTION
Pretrained large language models perform exceptionally well on a wide range of tasks, including language understanding (Chang et al. 2024 Nam et al. 2024), text generation (Zhao et al. 2023 Wang et al., 2024b), code synthesis (Jang et al. 2024 N guyen et al. 2025). ) and decision-making
(Ye et al. 2025 Lin et al. 2025). However, strong task performance alone does not ensure that these models behave safely or in alignment with human values (Dai et al. 2024 Li et al.) 2025 Shen main alignment methods have been developed. Supervised fine-tuning (SFT) adjusts a base model using human curatte crompt-respont pairs to shape behavior directly. RLHF (Bai et al.) [2022] Ouyang et al.) (002) follows two-step process: first, a proxy reward model is trained on human preference data to capture desired outcomes; second, the model is optimized against this reward using algorithms like PPO (Schulman et al., (017). RLOO (Ahmadian et al. 2024), or GRPO (Shao et al. 2024). By decoupling reward learning from policy learning, RLHF can leverage vast amounts of unlabeled data and generalize alignment to novel inputs, enhancing both safety and overall model capabilities.
al. 2023 Zhang al. (0255)). To address these concerns,
Work done during internship at Amazon. Corresponning Author.

---

Published as a conference paper at ICLR 2026
However, RLHF is susceptible to reward hacking (Gao et al. 2023 Yang et al. 2024). wherein the policy discovers shortcuts that maximize proxy rewards. such as by producing repetitive or formulaic content, without genuinely advancing the intended behaviors. Similar issues also arise in
inference-time strategies like Best-of-N (BoN) sampling (Gulcehre et al. 2023 Dong et al. 2023
Gui al. 2024). Prior research has investigated several directions to address this issue. One line
of work focuses on improving the reward function through ensemble methods (Coste et al. 2024
Eisenstein al. 2023; Zhang al. 2024a| Yan al. 2024, Ramé al. 2024 Zhang al. 2024b)
However, they often require training multiple reward models, making them resource-intensing and
less practical for applications. Another line investigates constrained policy optimization (Moskovitz
et al. 2024 Zhang et al.] 2024b; Liu et al.) 2024b| Zhang et al., 2024c] Laidlaw et al., 2024). but performance often unstable due to sensitivity to hyperparameter tuning. ODIN (Chen et al., 2024)
trains separate reward functions for quality and length; however, our results (Fig. 2 show that using length alone as a biasing factor fails to prevent reward hacking. Recently, GRM(Yang et al., 2024
Dai ct al.] 2025 incorporates text generation regularization into reward modeling and outperforms prior methods. However, the conflicting objectives of reward modeling and text generation cause training instability and sensitivity to the balancing weight (Appendix []. Moreover, existing studies focus on in-distribution evaluations, and the effectiveness of these methods in OOD settings, where training and test prompt-response pairs come from different distributions, remains unexplored.
In this study, we first show experimentally that state-of-the-art methods fail when prompts used during PPO and BoN are drawn from a distribution different from the training data. This highlights a critical limitation in the gencralization ability of current reward models under OOD settings. We hypothesize that BT model trained only on chosen/rejected labels remains biased and cannot distin- guish ine-grained quality differences. Inspired by recent multi-objective reward modeling methods (Wang et al., [0)4e[d), which leverage annotations for attributes like helpfulness, verbosity, and cor- rectness, we examine their potential to mitigate reward hacking. By learning from multi-dimennsinal supervision, multi-objective reward models (MORMs) capture nuanced distinctions response quality and compel policy models to improve across all attributes simultancously, making it harder to generate low-quality outputs that nonetheless score highly (Sec. (3.. While prior work focused on interpretability and steerability, the robustness of MORMs in policy learning remains underexplored.
Despite their potential, the performance of MORMs is constrained by the limited availability of large-scale, high-quality annotated data (Wang et al. 2024e d). This limitation arises either from the low-quality annotations produced by LLM Judge (Cui et al. 2023;; Kim et al.) 2023 Li et al.,
2024 Gu al. (024) or from the high-quality annotations produced by humans that are difficult to scale Wang et 2024f). A more detailed discussion of the data availability challenge is provided in [p Consequently, their scoring performance often falls behind that of single-objective reward models (SORMs) (Lambert et al. 2024; Liu et al., 2025b), which are typically trained on large-scale preference datasets with chosen/rejected labels that are easier to collect. A promising approach is to train a strong SORM and complement it with an MORM to enhance robustness against reward hacking. Though our empirical results in Sec. [ verify its promise, this approach faces two key issues: (1) it requires two independent inference passes, making the process computationally expensive in practice; and (2) the weaker performance of the MORM directly degrades the quality of the aggregated output, thereby becoming bottleneck in the overall system, as shown in Fig, 2 (d).
Thus, we study a novel problem: how to efficiently mitigate reward hacking in challenging OOD settings using tineigrained attribute scores, without additional costly multi-attribute preference data? To address this, we propose a simple, yet effective and theoretically grounded reward modeling frame- work, termed the Joint Single and Multi-Objecctve Reward Model (SMORM). The proposed SMORM framework addresses the first challenge by requiring only a single forward pass through a shared backbone. We theoretically establish the connection between the commonly used Bradley-Terry (Bradley& & Terry 1952) loss for training the SORM and the regression loss used for training the MORM. Our theoretical analysis and extensive empirical results demonstrate that: (1) training the multi-objective head refines the embedding space such that the representations capture quality distinctions across multiple attributes, thereby enhancing the generalizability of the single-ohjective head and improving its robustness against reward hacking.: and (2) training the single-objective head helps correct the positioning of responses in the embedding space, enabling the multi-objective head to perform competitively even with limited data. Overall, the joint training of both heads over shared embedding space leads to complementary benefits. Using the same multi- -bjti-objective dataset,
2

---

Published as a conference paper at ICLR 2026
SMORM enables a 7B model to outperform a 70B baseline model. Importantly, SMORM training is flexible in that the prompt-respons. pairs used to train the two heads do not need to be identical.
Our primary contributions are: (1) We empirically show that state-of-the-ar reward models suffer from reward hacking during PPO in OOD settings-a critical issue largely overlooked in prior work. (2) We introduce SMORM, a training framework that, while simple in form, is novel, non- trivial, and theoretically grounded. (3) This is the first work to provide theoretical analysis establishing principled connection between Bradley-Terry preference modeling and multi- objective regression, revealing their complementary benefits. (4) SMORM also tackles the key challenge of improving the performance of a multi-objective reward model without requiring additional, hard-to-obtaln data.
2 BACKGROUND
Bradley-Terry Single-Objective Reward Modeling. Single-objective reward modeling typically builds on the Bradley-Tere framework (Bradley & Terry 1952), which distinguishes between a chosen response ye and a rejected response y, for a given prompt æ. This is achieved by optimizing the following loss function:
ming Trewadd () = - (2 [log (a (ro(2, ye) - ro (z, B+))),,
where is the reward model parameterized by 0, (3(2,,)) v) denotes the reward score assigned by re for the output y given the prompt z, and (-)) is the sigmoid function. Minimizing this loss encourages the model to assign higher scores to outputs that are preferred by humans.
Multi-Objective Reward Modeling, In many practical settings, evaluating language model outputs requires considering multiple aspects such correctness, coherence, and verbosity. Single reward signals often fail to capture this complexity. To address this, multi-objective reward models (Wang al. 2024e 2023b 2024c) generate separate reward signals for different response attributes. The (s....))DD [| (z, y) - -rl where Ry is the multi-objective reward model and r E R-K denotes attribute scores (e.g., correctness, verbosity). Each dimension reflects specific quality, enabling more interpretable and steerable evaluations. However, existing work has not examined how such models relate to reward hacking.
model trained as: min minoC(())
Best-of-n Sampling (BoN), Given an input x, BoN (Gulcehre et al. 2023 Dong al., 2023 Gui
ct al. 2024) first draws a set Yzen of n candidate outputs from the policy model and then selects the one that maximizes the reward-model score. It can be applied either to improve outputs at inference time or to drive an iterative optimization procedure: (Bo = arg maxyeX'om ro (o y).
Proximal Policy Optimization (PPO). PPO is widely adopted method for RLHF in opti- mizing language models (Ouyang et al., 2022] SSiennon et al. 2020] [w et al. 2024). Us- ing proxy reward model rø, PPO refines the policy model Te by maximizing its score under the proxy reward while incorporating a KI. divergence penalty: max&<RLun()) =
E [By=x((1a)) (ro(2, v)) - A KL. (T)())))) || (1x))) where S is a training set of prompts and X 2 0 is a KL regularization that controls how much To deviates from the initial policy n...
Reward Hacking. Reward hacking occurs when a policy exploits flaws in the reward function, attaining high scores by overfitting to spurious patterns rather than accomplishing the intended task (Fu et al.) 2025 Yang et al. 2024 Gao et al. 2023). In PPO, this manifests when the policy achieves higher scores from proxy reward model but performs worse under more reliable, human- aligned golden reward model. Additional discussion of related work and comparisons with existing approaches are provided in Appendix A
SMORM: MITIGATING REWARD HACKING IN OOD SETTING
In this section, we conduct experiments to evaluate the robustness of existing SOTA methods and
proposed framework against reward hacking in both PPO and BoN under OOD settings,
more challenging setting largely overlooked in prior work (Yang et al., 2024 Coste et al. 2024
Eisenstein et al. 2023 Zhang et al. 2024al Yan et al. 2024 Ramé et al. 2024 Dai al. 2025
3

---

Published as a conference paper at ICLR 2026
(a) Inforence with Single Otjective Reward Model (c) Training of 3MORM (Ours)
Riverd Hocking
M Mipsiigg
Faod irference
Ofjectivn Head
Sironn Single and Multi
Hackiig
Chrosen Cod
Opamization
Single-Otjectivve Head
Milggding Reoorr Lxsking
TelRe.Jee
Singlo-Objectivee BT Loss
Kg
Applazer
Flow Infoemnce
Hobp deeee
Cohamroeci
MUNl-Okjective Regression Lose
Wleck
Hoad
Muti Oujooii Preferenca
Mull-Okjedivv Heed
(b) Joint Inference with Single and Multi-bkjective Reward Models
Figure 1: Illustration of SMORM training and its advantages over baseline methods. The training objective of SMORM is an original contribution, fundamentally differing from previous approaches and supported by theoretical analysis (see Appendix for detailed comparisons).
Intuitively, multii-bjecctiv reward model (MORM) evaluates response quality across multiple dimensions, forcing the policy to balance various attributes during generation. For example, it cannot maximize the overall score by optimizing helpfulness alone while neglecting verbosity. However, the performance of MORM is limited by the need for large-scale, high-quality multi-attribute labels, which are typically more difficult to collect than binary chosen/rejected labels (Wang et al.] (2024f). This motivates us to design ensemble framework that complements the stronger SORM with a MORM to enhance robustness against reward hacking, as shown in Fig. [b). However, this framework encounters two major issues: (1) it requires two separate inference passes, leading to significant computational overhead; and (2) the weaker performance of the MORM degrades the
quality of the aggregated output, becoming bottleneck for the system, verified in Fig. [
To overcome these limitations, we introduce a simple yet effective, theoretically grounded reward modeling framework called the Single and Multi-Objective Reward Model (SMORM). As shown in Fig. (c), SMORM jointly trains a BT single- and multi-objective reward function based on shared embedding space. addresses the issue of efficiency by requiring only a single forward pass through the backbone. Furthermore, training the single-objective function along the chosen/rejected dimension helps shape the embedding space and correct the positioning of samples within it, thereby enabling the multi-objective head to achieve competitive performance with limited data. We will theoretically and empirically justify this effect in Section
Specifically, given a pre-trained decoder-only LLM without the original output linear layer as the feature extractor fa. We pass x + y, the concatenation the hidden state of the final decoder layer as a d-dimensional feature. On top of fe. we attach two head with weights ws E Rdx1 which outputs a scalar rating, and a multi-objcctive head with weights WM E d which produces a k-dimensional vector of attribute scores. Given D3 = {I,, Yes yr} as the chosen-rejected preference dataset and Das = {zm. Dm r} the multi-attribute preference dataset, SMORM is trained with the following loss function:
and y. through the decoder layers and take
linear heads: a
min 8, w.w.
-Eps
[log a
(W ((o(se, - fo( yr + EDS [W (o vm) - x1/ (2)
Although the objective may superficially resemble a simple combination of BT and regression losses, their joint training is non-trivial due their fundamentally different forms. How these objectives influence one another has remained unexplored in prior work; we are the first to establish a theoretical connection and to show their complementary benefits (Sec. 4 The first head with weight ws outputs score along the chosen/rejected dimension, while the second head with weight wM outputs scores on multiple attributes (e.g. helpfulness, coherence and verbosity). SMORM supports multiple inference strategies: SMORM-F uses only the first head to produce the reward score, SMORM-L computes the mean of the scores from the second head, and SMORM-M averages the scores from both the first and second heads.
Experimental Setup. For training SMORM, we use Skywork80K (Liu et al., 2024a) as Ds and He Helpsteer2 (Wang et al.) 2024f) as Dar. To train the multi-attribute head using only label information without extra data or domain knowledge, we filter D& to retain only samples that also appear in Ds. We compare SMORM against several baselines, including: (1) Baseline Classifier, trained using the original reward loss defined in Eq.
T (2) ODIN, which trains two separate reward
functions for quality and length (Chen al. 2024 (3) Baseline SM, which trains baseline SORM
4

---

Published as a conference paper at ICLR 2026
and MORM separately and aggregates their results during inference and (4) GRM Yang et al. 2024 which incorporates supervised fine-tuning (SFT) regularization. We use gemma-2B- 22-it Team et al. 2024) the backbone model for all methods. In PPO experiment following (Yang et al. 2024).
we downsample 20K samples from the Un fied-FeFerrrr FFed-Fackkk dataset to optimize the PPO policy, reserving an additional 1K samples as a held-out test set for evaluation. Following prior work (Gao iet al. [023) Coste et al. 2023; Yang et al., (024). we perform BoN sampling on this evaluation set, selecting the best of n responses per prompt based on proxy model scores. The selected responses are then evaluated using the gold reward model, and gold scores are averaged across all prompts to assess true quality. We vary KL divergence from 0 to 5 by adjusting from 405, using the relation For both experiments, 1ommma-2B-15 serves as the policy model, and the gold reward model is a 7B human-preferenee model fine-tuned on the full Un fied Feedddeeebbck dataset.
KLBON log n
PPO (gemmo-2h-1)
PPO (gumma-2h-i)
BoN (gemmm-22--))
BoN (gammss-2t-H
0.8
Pr0 Score
Drosy Soom
CRV
Dositer
bypRyy Evlort
Ratalio
20
kandir
GEN
60
10000
20000
30000
50000
20000
30000
1
Trainig Samples
Training Samples
Divergence
KL Dierggrcce
(a)
(b)
(c)
(d)
Figure 2: Proxy and gold scores from (a)(b) PPO and (c)(d BoN experiments under the OOD setting, using gemma-2B-it 2B it as the base model. All rewards are normalized to start from O.
The results of the PPO and BoN experiments are presented in Fig. 2 (b) and (c)(d) respectively. From these figures, we observe the following: (1) Baselines Fail to Mitigate Reward Hacking. We observe that although GRM demonstrates increased golden scores during PPO, its average gold scores decrease as KL divergence increases in the BoN setting, suggesting reward hacking. ODIN shows a modest increase in golden scores under BoN, but a decrease during PPO, indicating that addressing length alone as a bias is insufficient to mitigate reward hacking effectively.
(2) Incorporating Multi- Attribute Scores Helps Mitigate Reward Hacking in OOD Setting. In both experimental settings, we find that the Baseline SM outperforms both GRM and ODIN, exhibiting a steady increase in golden scores as the proxy score increases. (3) Weak Multi-Objeetive Reward Functions Become a Bottleneck. While Baseline SM shows promise. its performance in the BoN experiment is even worse than that of the baseline single-objective reward function. This suggests that a weak multi-objective reward function Shows Superior Performance. Both SMORM-F and SMORM-M significantly outperform all baselines across both experiments. Notably, SMORM-F performs comparably to SMORM-M. To theoretically justify this phenomenon, we propose the following theorem:
be detrimental and become a bottleneck in the overall system. (4) SMORM
Theorem 1 (Implicit Multi-Attribute Effect). Let a reward model be trained under the SMORM framework, and suppose the following conditions hold: (1) Bounded features: There exists
B < oo such that [] y) < B for every (x, y)- (2) Posifive-definite covariances: let fe fe(xe.ye) fr fo vr), fm = fe(Im.ym)) Es == ED lffe fr) matrices. (3) Positive correlation: Let us == Ei We [J(( - (x.. (x.. and let Cm == (mm)) [o (I ym) ) r E RUXk
J and EM := ED fm] are positive-defint
Then a == MS2 Cas has non-negative sum, i.e. 1 1 Z O. As the optimization of both re- ward heads converge to their population minimizers, there exist constants c
1
and
K (n3 Σg us)
E 2 0 dedendding only on B and second-order moments such that for every pair y):
Tm(x,y) B)
1 S (T,,fez. > c (w]]oo(a, v)) - -E= cr cr,
(3)
A generalized theory that explicitly accounts for data size is presented in Theorem 3 in the Appendix.
We provide detailed justifications of the assumptions and the proofs in the Appendix D.2]
The
Fcedbaack
5

---

Published as a conference paper at ICLR 2026
theorem is proved under the assumption that the aggregated attribute scores in the multi-objectiin preference data are positively correlated with the score provided by a pretrained single-obeective reward function Ws. This is well-justified: ws can be trained on large-scale, high-qualing preference data and thus provides reliable scoring. Moreover, multi-objective annotations typically follow the principle that chosen responses should have higher aggregated scores than rejected ones, and thus positive correlation can be reasonably expected. Notably, this assumption is general and does not require the training data for the two heads share the same prompt-respone pairs domain.
From this theorem, we derive two conclusions: (1) If a response achieves a high single-objective score rs 2 T, its mulli-attribbut average score I'm lower-bounded by CT E. This result implies that even when the multi aulti--ttribute head is ignored, high single-objective score alone ensures respectable level of fine-grai fine-grained quality. This explains why policies trained with SMORM-F attain performance comparable to those trained with SMORM-M. (2) For any two responses ya and ys to the same prompt z, if r, (a. yA) > ra (a, VB), then follows that cr, MA) - cr, (r, yB) E. That is, higher single-objective score implies a strictly higher lower bound on the corresponding multi-attribute score Tm. This enables training a strong single-objective function from abundant preference data and leveraging it to guide the multi-objective function, without the need for costly fine-grained multi- attribute annotations. Empirical results in Sec. 5.2 confirm this conclusion.
4 SUPERIOR SCORING PERFORMANCE OF SMORM
We have empirically and theoretically demonstrated that SMORM effectively mitigates reward hacking, even under challenging OOD scenarios. Although Theorem [ establishes that SMORM can implicitly leverage stronger single-objective reward function to guide the multi-objective one, potential concern remains: joint training may degrade the performance of that single-objective function. To address this concern, we begin this section by introducing a lemma that connects Bradley-Terry reward modeling with multi-objective reward regression. We then present theorem demonstrating that SMORM improves the performance both reward functions by learning a more effective feature extractor. Unlike ODIN (Chen et al., 2024). which trains reward functions for both quality and length using the BT loss, the interaction between BT loss and MSE regression loss when sharing common embedding space is not straightforward to characterize. Therefore, we propose the following lemma to bridge this gap.
Lemma 1 (Pairwise Preference Error to MSE Loss), Let YA, UB be pair of responses. Assume gs (y) is the ground truth score and r, (y) is the predicted score under a Bradley-Terry model. Then:
P( > WB) = a (T.(NA) (uB)), P' > UB) a (Is(UA) - (s()))
where (t) = 14e-c- The expected preference error satisfies:
1
EDS [P(yA > yn) P" (yΛ yn)l 4 Bos (V2MSE(T.))
with MSE(r.) (re(v) (x)))
Lemma 1 shows that the expected BT loss is upper bounded by the MSE, i.e., the square of the regression loss. With this lemma, we propose the following theorem:
Theorem 2. Under the same assumptions as in Theorem | and assuming that the feature extractor La is differentiable, let O denote the maximum likelihood estimator (MLE) of the ground truth optimal parameter 8*. Let 0, and 0m denote the maximum likelihood estimators of the single- and multi- objective reward functions, respectively. Define Ms(y) = w g fo- (y), MM((y) w Jo (y). Then, for a response y, the mean squared error (MSE) of the predicted reward can be approximated as:
MSEş & VoMs(u) Cov (8.) voMs(y) + Too, MSEM zo VoMa((y) Cov (B... VoMM(()) + ooo.
where doo is the intrinsic randomness in the label. Moreover, SMORM yields lower asymptotic MSE for both the single- and multi objectve heads compared to training either head alone:
MSE.SMORM
MSFningle
MSE
MSFmulii
(5)

---

Published as a conference paper at ICLR 2026
Detailed proofs of the above lemma and theorem provided in Appendix D By Lemma Theorem 2 a reduction in MSE directly tightens the bound on pairwise preference prediction error. This is the first theoretical guarantee that shared BT-regression architecture is strictly superior to training the two heads independently, a connection never explored in prior work. See Appendix for details highlighting the originality and significance of our theoretical contribution.
I and
EXPERIMENTS
In this section, we present comprehensive evaluation of SMORM. We conduct experiments under both in m--istribution and out-of outoof-distribution settings to assess how SMORM improves scoring capa- bility and mitigates reward hacking. Our results demonstrate the following advantages of SMORM: (1) Enhancing scoring performance. SMORM significantly improves the scoring capability of the multi-objective reward function without requiring additional multi-attribute preference data, which are often difficult to collect (Sec. 5.2). (2) Mitigating reward hacking. SMORM substantially enhances the robustness of the single lobjecobvtvvvveeee reward function against reward hacking in both ID and challenging OOD settings (Sec. (.3), while also improving its scoring performance (Sec. 5.2). (3) Flexible training. SMORM does not require the single-objective dataset Ds and the multi-objective dataset DM to share the same prompt-response pairs, allowing for a more flexible training process.
5.1 EXPERIMENTAL SETUP
Datasets and Benchmarks. In Section 5.2] we use the Unified- Sniifie-FFedbacc Feedback dataset the single- objective preference dataset Ds. one of the largest collections of pairwise human preferences. To assess robustness across data scales, we evaluate two settings: one with 400K samples from Ds and another with 40K. For the multi-objective dataset Dar, we use Fetaaeeeeebbck (Cui al. (023) in the 400K setup, which includes 240K GPT-annotated prompt-response pairs with fine-grained attribute scores. In the 40K setup, we use HelpSteer2 (Wang et al., 2024f). high-quality human- annotated dataset with 20K samples. SORM is trained only on Ds. MORM on DM. and SMORM jointly both. The trained reward models are evaluated on RewardBench (Lambert et al.) 2024) and R.M-Bench (Liu et al.) (0256). In the RLHF experiments (Section 5.3). we consider both ID and OOD settings. For ID, we downsample 20K samples from Uni SenlfieddFeedbaee
D3 to train
reward models and the PPO policy, For OOD, we use Skywork80K (Lu et al. 2024a) as Ds. In both cases, He 1pSteer2 serves as DM, and IK samples from Uni for policy evaluation.
are reserved
Base Models. Following (Yang al. 2024), we adopt gemma-2B-it (Team al. 7B- IB-Instruct-vO.2 (Jiang et al.) 2023) as the base models for preference learning. For the RLHF experiments, gemma-2B-ii serves as the policy model in both the BoN and PPO settings. The gold reward model is 7B human preference model fine-tuned on the entire Chified-Feedbbce dataset.
2024) and Mistral-
Baselines. We adopt representative and SOTA baselines: (1) Baseline Classifier, trained using the
original reward loss as defined in Eq. D (2) Margin (Touvron et al. (023), which augments the original reward loss with an additional margin term; (3) Label Smooth (Wang et al. 2024a). which addresses overfitting by penalizing overconfident predictions: (4) Ensemble, which combines the outputs of three reward models by taking either the average or the minimum score (Coste et al. 2002);; ; and (5) GRM(YYng et al., 2024) with two types of regularization: GRM w/ dpo and GRM w/ sft.
Details on baselines and implementation are in Appendix !
RLHE The training and evaluation pipeline of PPO and BoN follow the setting in Sec.
5.2 EVALUATION ON REWARD MODELING
Comparison to Single-Objeetivy Reward Model. Table compares SMORM-F with baseline SORMs RewardBench using gemma-2B-it as the base model; results with Mistral-7B-Insttee are reported in Appendix C.6 SMORM-F consistently achieves the highest average performance across all settings, supporting Theorem 2] which shows that joint training yields superior performance over training ougle--obeective reward functions in isolation. Additional OOD evaluation results for SMORM-F are provided in Appendix G
7

---

Published as a conference paper at ICLR 2026
Table Comparison of SMORM-F and baselines on RewardBench.
Da/Day ay: UnifledFeedbacc 400kUUaaaeeelaackk
Dy/Dur Dur: UnifieFFeedback 40k/HelpSteer?
Reward model
Chat
Chat-Hard
Safety
Reasoning
Avg
Chat
Chat-Hard
Safety
Reasoning
Avg
Baseline (Single)
95.5
38,0
73.8
65.3
68.2
94.7
37.5
66.2
58.4
64.2
Baseline margin
95.8
38.4
73.9
72.5
70.2
97.2
37.5
56.8
72.7
66.1
Label smooth
94.4
37.3
73.2
77.4
70.6
91.6
39.0
60.2
61.1
Ensemble
98.0
37.5
77.3
71.3
71.0
96.1
38.2
58.8
67.6
65.2
GRM (lincar) w/ dpo
96.7
39.0
76.4
68.5
70.2
94.7
38.4
62.5
51.2
61.7
(linear)
71.5
77.0
GRM w/ dpo
95.8
40.1
78.7
66.2
70.2
92.5
39,9
72.5
61.4
66.6
GRM w/ sft
97.8
42.1
77.9
65.2
70.8
94.1
41.9
69.5
61.5
66.8
SMORM-F
45.5
70.9
72.8
44.1
62.7
71.0
Table 2: Comparison of SMORM-L and baseline MORM on RewardBench and RM-Bench
Reward model
Chat
Chat Hard
Safety
Reasoning
RewardBench
RM-Bench
Base Model: Gemma 2b it, Ps/Dar: UnifiedFeedback 400k/UnraFcedbuck
Baseline (Multi)
64.2
50.0
46.1
42.3
50.6
50.2
SMORM-L
90.8
48.3
61.5
53.7
63.6
55.1
Base Model: Gemma Ds/Dat /Dr:: UnifiedFeedback 40k/HelpSteer2
Baseline (Multi)
84.1
39.0
57.4
42.6
55.8
51.8
SMORM-L
94.9
39.3
75.8
51.4
65.4
54.1
Base Model: Mistral 7b Instruct, Ds/Dar UnifiedFeedback 400k/Utrraeedhacck
Baseline (Multi)
95.5
65.1
75.9
76.3
57.2
SMORM-L.
97.8
61.0
86.4
77.7
80.7
62.6
Base Model: Mistral 7b Instruct, Ds/DM: UnifiedFeedback 40k/HelpSteer2
Baseline (Multi)
71.8
60.1
54.3
78.0
66.0
52.0
SMORM-L 94.4
61.8
83.6
79.7
79.9
64.4
Comparison to Multi-Objective Reward Model. Table compares SMORM-L with baseline MORMs RewardBench and RM-Bench. With Mistral-7B-Insstee as the base model and 40K samples from Uni FiedF eedback, SMORM achieves gains of +13.9 on RewardBench and +12.4 on RM-Bench. These results validate Theorem which shows that BT modeling corrects response positioning so that the multi-objective score is lower-bounded by the single-objective score, and further support Theorem confirming that SMORM produces reward models that consistently outperform their baselines.
Comparison to Advanced Reward Models. We follow (Wang et al. 2024d) and train gating network that assigns weights to each attribute during inference. Our 7B model is initialized from Mistral-7B-IInstrett-vo
(Jiang et al. 2023) and trained on Ds/Da: Uni edFeedbacc (40K) and He HelpSteer2 (20K). The 8B model is initialized from Llama-3.1-80 Instrucc (Grattafioni ct al. 2024) and trained on Skyworkä OK (Liu et al.) 2024a) and HelpSteer2 (20K) both D-- We compare against large multi-objective reward models trained on He H pSt.err r with tuned attribute weights (Wang et al., (202f)), and MemoRRM-LLama3-BB-e
cases, the gating network is trained with BT loss
(Wang al. (024d). which is trained far more data: 585.4K samples for its multi-objective head and 1,004.4K for its gating network, versus only 20K, 40K, and 80K for ours. As shown in Table our model underperforms the 340B baseline but surpasses the 70B model. Remarkably, our 8B model matches
3-.-v0.. despite using 15.9x less data.
Table 3: Comparison with Advanced Multi- Objective Reward Models on RewardBench.
Reward model
Size
Model Size
Chat
Chat Hard
Safety
Reasoning
Avg
Nemotmon 4-340B RM (Wang al 20244
20K 585 4K
340B
92.0
95.8
87.1
91.5
93.7
AmmoRM-Llmma-9B-. Wang et ah 2024d
8B
96.9
76.8
90.5
97.3
90.4
Llama-3-70B--e (Wang 20241
20K
70B
91.3
80.3
92.8
90.7
88.8
SMORM-L 7B (Ours)
20K
7B
95.0
80.5
91.6
89.0
89.0
SMORM-I. (Ours)
20K
8B
94.7
85.1
90.5
91.3
90.4
8

---

Published as a conference paper at ICLR 2026
PPO (gemma 2b
FPO (gemma
PPO (Mstral78 7B
PPO (Mstra 7B
10
Baveine
Frodr Scote
000 Spote
S00s 2004
Golden Spoore
2.0
0.
DVOTH M4
DVOTIV 14
GRU
02
33000
20000
20030 Training Samples
20006
12000 20000
30003
80030
12000
20003 20030 8000 Traiing Samples
Treenin Samples
Traiiig Samples
(a)
(b)
(c)
(d)
Figure 3: Proxy scores and gold scores of PPO experiments for reward model based on (a)(b)
gemma2b-it and (c)(d) Mistral-7B -instal-7--Instruet.. All rewards are normalized to start from 0.
(gemma-2h--i)
(gemma-2bb-))
(Niorral-7B-ttttturrr
(Mistne-70-mmmuuc)
Guedrr
CBM
Prony Score
Golden Store
0.4
5005 Aold
100 2-4
Jandia seory
1 V
CPM SVCRBAV
KL Divergence
KL Divergence
KL Divergenes
KL Divergence
(a)
(b)
(c)
(d)
Figure 4: Proxy scores and gold scores of BoN experiments for base models of (a)(b) gemma-2b-it and (c) Mistral-7B-IIstruue Rewards are normalized to start from O.
5.3 EVALUATION ON RLHF
In-Distribution Setting. Figures 4n the results of PPO and BoN under the in-distribution setting. For PPO, we observe that the gold score of the baseline reward function increases slowly in the early stages of training but subsequently declines, while the proxy score continues to rise. This pattern indicates reward overoptimization. In contrast, GRM exhibits rapid initial increase in gold score, followed by similar decline trend. Notably, both SMORM-F and SMORM-M show a consistent increase in gold score throughout training. SMORM-F achieves performance comparable to SMORM-M, which further supports Theorem D regarding the implicit benefit of joint training. In the BoN experiments, when using gemma 2B it as the base model, we find that the gold score of the baseline reward function decreases as the KL divergence increases. For GRM, the gold score increases slowly and then plateaus. Both SMORM-F and -M show consistent increase in gold scores across KL ranges, demonstrating the robustness of our framework in the ID setting.
Dut-of-Distributtion Setting. Figure 5 presents PPO and BoN results under the OOD setting using -istral-7B-IIstrucc as the base model. Results with gemma-2B-ir were shown earlier in Section 3 While the baseline reward function and GRM do not exhibit clear reward hacking with the stronger model in the OOD setting, the performance gap between SMORM and GRM becomes more pronounced compared the in-distribution results in Figure This underscores the limitations of existing methods and demonstrates the robustness of SMORM, especially under challenging OOD conditions. Additional results comparing PPO- optimizee models are provided in Appendix [H]
6 CONCLUSION
In this paper, we propose effective and theoretically grounded method that jointly trains Bradley-Terry reward function and a multi-objective reward function using a shared embedding space. Our theoretical analysis and extensive empirical results demonstrate that this joint training approach enhances the robustness of the BT function against reward hacking and significantly improves the scoring performance of the multi-objective function, revealing their complementary benefits.

---

Published as a conference paper at ICLR 2026
PPO (Mistral-B
(Mistaal
(Mistral
BB-inctuct
0.3
Dovire
Seusine
Proiy Store
Svonsr
Gorden
Golden
ofly DAONUM
SMM
19330
2003
2300
1000e
20000
20000
Training Sampls
Tranting Samples
KL Diveggnoe
Diviegenoe
(a)
(b)
(c)
(d)
Figure 5: Proxy and gold scores from (a)(b) PPO and (c)(d BoN experiments under the OOD setting.
ACKNOWLEDGMENT
This material is based upon work supported by, or in part by the Army Research Office (ARO) under grant number W91 1911N--21-00199 and the Department of Homeland Security (DHS) under grant number 17STCIN00001 -05-00. The views and conclusions contained this material are those the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the funding agencies.
ETHICS STATEMENT
This work does not involve human subjects or the collection of new datasets. Experiments use established corpora and benchmarks under their licenses.
REPRODUCIBILIIY STATEMENT
We provided a detailed description of experimental settings in Sec.
REFERENCES
Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340b technical report. arXiv preprint arXiv:2406. 1174, 2024.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402. 14740, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05800 05862, 2022.
Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138 -56, 2006.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1-45, 2024.
Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad bi, and Bryan Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf. In International Conference on Machine Learning, pp. 7935-7952, PMLR, 2024.
10

---

Published as a conference paper at ICLR 2026
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating Ilms by human preference. In Forty-first International Conference Machine Learning, 2024.
Thomas Coste,
Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint urXiv:2310.0274, 2023.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In The Twelfth International Conference Learning Representations, 2024.
net http://teeeooootooedddoeottttt/e
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. 2023.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024. URL https:tpp//.omc. http::://oppeeeveeee nww.forueeeeumuummmeeee form??
id=TyF
Juntao Dai, Taiye Chen, Yaodong Yang, Qian Zheng, and Gang Pan. Mitigating reward over- optimization in RLHF via behavior-supported regularization. In The Thirteenth International Conference on Learning Representations, 2025, /RLhhttss:/// /ttp///ppeeeeeeeeeeeeeeeeee net/ forum?
1s
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions Machine Learning Research, 2023. ISSN 2835-8856.
URL httpp:://pppeeeeeaaeap/
net wwrum? www.eeeeooidddddn/eneaaaeaaa
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arxiv:2405.000500 2024.
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' D'Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herd- ing? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09224,, 2023.
Jiay Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang, Qi Han, and Yanghua Xiao. Reward shaping to mitigate reward hacking in rlhf. arXiv preprint 2rXiv 2025.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10866. PMLR, 2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuchao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, al. A survey on alu-ae---uudge. arXiv preprint 1rXiv:2411.15555 155994 2024.
Lin Gui, Cristina Garbacea, and Victor Veitch. BoNBon alignment for large language models and the sweetness of best-of-n sampling. In The eiiht Annual Conference on Neural Information Processing Systems, 2024. htt.:tttt:////ttttttttt /ttp:/ttppss//peeeeeeeee.e. net/ wwwp:/eneeettoooideeeeeeeemm www.:ruummidhhaaaMMiMa rbx5
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308. 08998, 2023.
Geyang Guo, Ranchi Zhao, Tianyi Tang, Wayne Xin Zhao, and Ji-Rong Wen. Beyond imitation: Leveraging fine-grained quality signals for alignment. arXiv preprint arXiv:2311.04077 2023.

---

Published as a conference paper at ICLR 2026
Jiwoo Hong, Noah Lee, Eunki Kim, Guijin Son, Woojin Chung, Aman Gupta, Shao Tang, and James Thome. On the robustness of reward models for language model alignment. arXiv preprins arXiv:2505.07271, 2025.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https: //axxiv org/abs/ 20t./3330.00000 06825
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey large language models for code generation. arXiv preprint arXiv:2406.00515. 2024.
Steven M Kay. Fundamentals of statistical signal processing: estimation theory. Prentice-Hall, Inc., 1993.
Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment reward-guided search. arXiv preprint arXi::2402.01694, 2024.
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. Prometheus: Inducing fine-grained eval- uation capability in language models. In The Twelfth International Conference on Learning Representations, 2023.
Cassidy Laidlaw, Shivam Singhal, and Anca Dragan. Correlated proxies: A new definition and improved mitigation for reward hacking. arXiv preprint arXiv:2403.03185, 2024.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannanch Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024.
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges
auddse- arXiv preprint arXiv:2411.16594, 2024.
Xiaomin Li, Mingye Gao, Zhiwei Zhang, Jingxuan Fan, and Weiyu Li. Data-adaptive safety rules for training reward models. arXiv preprini 2501:::5001155555 2025.
Minhua Lin, Hui Liu, Xianfeng Tang, Jingying Zeng, Zhenwei Dai, Chen Luo, Zheng Li, Xiang Zhang, Qi He, and Suhang Wang. How far are lims from real search? comprehensive study on efficiency, completeness, and inherent capabilities. arXiv preprint arXiv:2502. 18387, 2025.
Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, al. Mitigating the alignment tax of rlhf. arXiv preprint arXiv:2309.06256. 2023.
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.18455 2024a.
Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin. Tianhe Yu. Daniel Sohn, Anastasia Makarova, Jeremiah Zhe Liu, Yuan Liu, Bilal Piot, Abe Ittycheriah, Aviral Kumar, and Mohammad Salch. RRM: Robust reward model training mitigates reward hacking. In The Thirteenth International Conference Learning Representations, 2025a. URL htttstts::/ http::///ppeeeeeeeeecoomm. net. forum? 1d=
Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. RM-bench: Benchmarking reward models of language models with subtlety and style. In The Thirteenth International
Conference on Learning Representations, 2025b. URL htt:////ppe//ei//ee////e/e/ wwt/ www.re//owwweeeeeoouutm Id=0EHrm@PEdd)
URI/sttps://opeeeeiiee
12

---

Published as a conference paper at ICLR 2026
Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang. Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in RLHF: Your SFT loss is implicitly an adversarial regularizer. In The ehirty-eigtth Annual Conference on Neural Information Processing Systems, 2024b. URL /ttp::////ppeeeeeeeeeeeeeee net. fante/oorune forum? i 2=2cc3 1Phke0
Feng Luo, Rui Yang, Hao Sun, Chunyuan Deng, Jiarui Yao, Jingyan Shen, Huan Zhang, and Hanjie Chen. Rethinking diverse human preference learning through principal component analysis. arXiv preprint ar arXiv:2502. 13131, 2025.
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.0m 06592, 2024.
Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. Inform: Mitigating reward hacking in rlhf via cotommtiion-thorree reward modeling. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
Ted Moskovitz, Aaditya K Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca Dragan, and Stephen Marcus McAleer. Confronting reward model overoptimization with con- strained RLHF. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrest
net forum id g
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vincet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332. 2021.
Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. Using an IIm help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference Software Engineering, pp. 1-13, 2024.
Dung Manh Nguyen, Thang Chau Phan, Nam Le Hai, Tien-Thong Doan, Nam Nguyen, Quang Pham, and Nghi D. Q. Bui. CodeMMLU: A multi-task benchmark for assessing code understanding & reasoning capabilities of codeLLMs. In The Thirteenth International Conference on Learning Representations, 2025.
net htwp::/wwweeeeoee.eeeemmmm (tw.orrummmid-ChhhcccccC
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Silviu Pitis, Ziang Xiao, Nicolas Le Roux, and Alessandro Sordoni. Improving context-aware preference modeling for language models. Advances in Neural Information Processing Systems, 37:70793-70827, 2024.
Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personalizing reinforcement learning from human feedback with variational preference learning. Advances in Neural Information Processing Systems, 37:52516-522 52544, 2024.
Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. arXiv preprint arXiv:2401.12218 2024.
Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D Lee, and Sanjeev Arora. What makes reward model a good teacher? an optimization perspective. arXiv preprint arXiv:2503.15477, 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:/707.066 1707.06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03330 03300 2024.
13

---

Published as a conference paper at ICLR 2026
Jingyan Shen, Jiarui Yao, Rui Yang. Yifan Sun, Feng Luo, Rui Pan, Tong Zhang, and Han Zhao. Micro: Mixture modeling and catext-awarn routing for personalized preference learning. arXiv preprint arXii:2505.
2025.
Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. Large language model alignment: A survey, arXiv preprint arXiv:2309.15000 15025, 2023.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 01:3008-30214. 2020.
Hao Sun, Yunyi Shen, and Jean-Francois Ton. Rethinking reward modeling in preference-based large language model alignment. In The Thirleenth International Conference on Learning Representa- tions, 2025. URL
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08225 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Ilama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307 002888, 2023.
Binghai Wang, Rui Zheng. Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, al. Secrets of rlhf in large language models part i:: Reward modeling. arXiv preprint arXiv:2401.0608) 2024a.
Chaoqi Wang, Zhuokai Zhao, Yibo Jiang, Zhaorun Chen, Chen Zhu, Yuxin Chen, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Hao Ma, et al. Beyond reward hacking: Causal rewards for large language model alignment. arXiv preprint arXiv:2501.0960 2025a.
Fali Wang, Zhiwei Zhang, Xianren Zhang, Zongyu Wu, Tzuhao Mo, Qiuhao Lu, Wanjing Wang, Rui Li. Junjie Xu, Xianfeng Tang, et al. comprehensive survey of small language models the era of large language models: Techniques, enhancements, applications, collaboration with Ilms, and trustworthiness. arXiv preprint arXiv:2411.033350 2024b.
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of Ilms for diverse user preferences: Directional preference alignment with multi-objective rewards. In ACL, 2024c.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 10582-10592, 2024d.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce lims step-by-stey without human annotations. arXiv preprint arXiv:23/2.08893, 2023a.
Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, et al. Helpsteer: Multi-attribute helpfulness dataset for steerlm. arXiv preprint arXiv:2311.09528 2023b.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng. Gerald Shen, Daniel Egert, Jimmy Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open- dataset for training top- nop-performing reward models. In The Thirty-eight Conference on Neural Information Processing Systems Dutasets and Benchmarks Track, 2024e. URL AtR.htts::/ /oopnreviiii net forum?
1d=PwVKUF haNy
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev, Helpsteer2: Open-source dataset for training lerfoernognnng reward models, 2024f.
14

---

Published as a conference paper at ICLR 2026
Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. Helpsteer2-preference Complementing ratings with preferences. In The Thireentth International Conference Learning Representations, 2025b. | https:
//oenreview www.revview... net wtt.r//wweee///oemmmm wtw.rrrrmmmmi...m..PPPPPpp...
Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In International conference on machine learning. pp. 23631122364, PMLR, 2022.
Xueru Wen, Jie Lou, Yaojie Lu, Hongyu Lin, Xing Xinyu Lu, Ben He, Xianpei Han, Debing Zhang, and Le Sun. Rethinking reward model evaluation: Are we barking up the wrong tree? In The Thirteenth International Conference on Learning Representations, 2025. htttst
/oopnnevveww.no net//
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-an natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38-45, 2020.
Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, and Jiantao Jiao. Pairwise proximal policy optimization: Language model alignment with comparative RL. In First Conference on Language Modeling, 2024. URL https:ttt//tt..... httt:::://pperevieee net/ forum?id=
71aA 11V2H
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36:59008-500 59033, 2023.
Yuzi Yan, Xingzhou Lou, Jialian Li, Yiping Zhang, Jian Xie, Chao Yu, Yu Wang, Dong Yan, and Yuan Shen. Reward-robust rlhf in Ilms. arXiv preprint arXiv:2409. 15360, 2024.
Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang. and Tong Zhang. Regularizing hidden states enables learning generalizable reward model for LLMs. In The ehihty-eighth Anmual Conference on Neural Information Processing Systems, 2024. BtLphttps: /oppr///eee/eeeeeeeeeeee hwt.:/wwwww.ee...eeeeeeeeeeeeeee forum?
1d= 9MHEMfY
Yining Ye, Xin Cong, Shizuo Tian, Yujia Qin, Chong Liu, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Rational mekision agent with learning internal utility judgment. In The Thirteenth
International Conference on Learning Representations, 2025. URL btttp::/opeerreviee
net Det/Fforum:11d-GBBXXYK2OC
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing Ilm reasoning generalists with preference trees. arXiv preprint 240::24444000778 2024.
Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen, Zhiqing Sun, and Chuang Gan. Improving reinforcement learning from human feedback with efficient reward model ensemble. arXiv preprint arXiv:2401. 16635, 2024a.
Xiaoying Zhang, Jean-Francois Ton, Wei Shen, Hongning Wang, and Yang Liu. Mitigating reward overoptimization via lightweight uncertainty estimation. In The Thirty-eighth Annual Confer- ence on Neural Information Processing Systems, 2024b. URL htLhsts:: htttps:://penneeeee..... Forum?id=kYio::t
Xiaoying Zhang, Jean-Francois Ton, Wei Shen, Hongning Wang, and Yang Liu. Overcoming reward overoptimization via adversarial policy optimization with lightweight uncertainty estimation. arXiv preprint 4rXiv:2403.05117 2024c.
Zhenru Zhang, Chujie Zheng. Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.03301, 2025a.
15

---

Published as a conference paper at ICLR 2026
Zhiwei Zhang, Fali Wang. Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpeng Yin, and Suhang Wang. Catastrophic failure LLM unlearning via quantization. In The Thirteenth International Conference Learning Representations, 2025b. URL htttaa://ppeeeeeieer ner//foum? d== HSeDYamnz
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. survey of large language models. arXiv preprint arXiv: 1(2), 2023.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving Ilm helpfulness harmlessness with rlaif. 2023, 2023.
Banghua Zhu, Michael I Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf. arXiv preprint arXiv:2401.16335 2024.
16

---

Published as a conference paper at ICLR 2026
A RELATED WORK AND KEY DIFFERENCES FROM PRIOR APPROACHES
In this section, we provide additional related work and clarify how SMORM differs from existing
approaches.
Reward Modeling. Reward models are designed to produce preference-aligned signals that guide the behavior of language models. In the context of large language models (LLMs), they act proxies for human preferences, providing feedback to the policy model during the alignment process (Ouyang
et al.
2022 Bai al. 2022 Dong et al. 2024 Wen et al. 2025 Dong et al. 2024 Razin et al. 2025 Lin et al. 2023 Sun et al. 2025). These models are typically constructed by attaching a classification
head a pretrained LLM, allowing it assign scores responses conditioned on prompts (Zhu et al.
2024). To ensure alignment with principles such as helpfulness, harmlessness, and 2023 Adler et al.
honesty, reward models are fine-tuned using human preference datasets (Wu al., [023 Guo et al.
2023 [ai et al. [2024] Chiang et al. [2024; Nakano et al. 2021). The resulting reward signals are then
used in policy optimization, enhancing LLM performance on complex downstream tasks, including
mathematical reasoning (Shao et al.
2024 Wang et al. 2023a Luo et al. 2024 Zhang et al. 2025a
Mitigating Reward Hacking in RLHE Reward hacking in reinforcement learning from human feedback (RLHF) arises when the policy model exploits imperfections in the reward model, thereby failing to learn the intended behaviors. A variety of strategies have been proposed to address this issue. One line of work focuses on enhancing the reward function using ensemble methods (Coste
cct al. 2024; Eisenstein et al. 2023 Zhang et al.) 2024a Yan et al., 2024; Ramé et al.) 2024] Zhang ct al. 2024b). While effective, these approaches typically require training multiple reward models, making them computationally expensive and less feasible for real-world deployment. Another
approach investigates constrained policy optimization (Moskovitz et al. 2024] Zhang et al., (2024b)
Liu et al. 2024b Zhang et al. 2024c; Laidlaw et al. 2024] Zhu et al. 2024p. However, these methods often suffer from performance instability due their sensitivity to hyperparameter tuning. More recently, GRM (Yang et al. 2024 Dai et al. 2025) incorporates text generation regularization into reward modeling and achieves superior performance compared to prior methods. Nonetheless, the inherent conflict between reward modeling and generation objectives introduces training instability and increases sensitivity to the choice of balancing weights.
Multi-objective reward modeling. (Pitis et al. 2024) demonstrates that reward models benefit from explicitly encoding conversational or task-specific context, mitigating preference inconsistencies under varying prompt conditions. (Shen et al.) (025) introduces mixture-of-experts routing to capture heterogeneous user preferences, highlighting that a single global reward model often fails to generalize across diverse human behaviors. (Luo et al.] (0255) proposes a PCA-based analysis of preference directions, showing that many reward-mode failures stem from collapsed or poorly estimated preference subspaces. (Poddar et al. 2024) adopts a probabilistic generative framework to model user-specific latent preference factors, enabling personalized RLHF policies that better reflect individual annotators' reward structures. These works collectively underscore that preference distributions are often multimodal and context-dependent, and they motivate our focus on analyzing robustness and over-optimization in reward modeling under prompt and model distribution shifts.
ORIGINALITY AND NOVELTY OF SMORM
ODIN (Chen et al., 2024) is a framework that trains two reward functions for response quality and length, sharing common embedding space. However, our work differs substantially from ODIN in several key aspects: (1) While ODIN uses the BT loss for both heads, we theoretically establish a connection between BT loss and multi-attribute regression loss. This advancement enables the integration of multiple fine-grained attributes beyond just response length. (2) Our work explicitly investigates policy optimization with PPO in -ut-of-distriuution (OOD) settings, a scenario largely overlooked by existing studies. Our empirical results in Sec. 3 also demonstrate that ODIN fails in this setting. (3) Whereas ODIN focuses solely on mitigating reward hacking, we additionally uncover complementary benefit: training single-objective reward function significantly enhances the scoring capability of the multi-objective reward model.
RRM (Liu et al. 2025a) is a general framework that employs causal approach to learn preferences independent of spurious artifacts via data augmentation during reward model training. However, (1) the data augmentation strategy significantly increases the computational cost of training reward
17

---

Published as a conference paper at ICLR 2026
model; (2) it remains unclear how adapt such augmentation methods to multi-objective reward modeling, especially when labels consist of fine-graing -e--grrined scores; and (3) its effectiveness in mitigating reward hacking in RLHF settings has yet to be empirically validated. Similarly, (Wang et al.] (225a) propose causal reward modeling approach that incorporates causal inference to reduce spurious correlations. InfoRM (Miao et al.) 2024) introduces variational information bottleneck objective filter out irrelevant information during reward modeling, However, similar to GRM, this objective is fundamentally at odds with the goal of accurate reward modeling, and the performance of InfoRM on RewardBench has not been evaluated. In contrast, beyond addressing reward hacking, a key contribution of our work is enhancing the performance of weak multi-objective reward models without requiring additional preference data.
Our SMORM introduces significant methodological and empirical innovations compared to (Wang et al. 2025b), for the following reasons:
Significant Differences in the Training Paradigm. The training paradigm in (Wang al. 2025b) is sequential: either initializing BT reward model from a regression reward model or vice versa. In contrast, SMORM adopts joint training paradigm, where both the BT and regression reward functions are optimized simultaneously at each training step. This co-training leads to synergistic learning and mutual improvement between the two reward heads, rather than a one- directional transfer.
(2) Significant Differences in Empirical Discoveries. and contrast them with our work below:
We summarize the empirical findings of
Method
BT Regres- sion
Regression BT
Co-Training
Paper
Wang
al.
Worse than dom (93.0 + 92.2)
Better than ran- dom (91.5 + 92.7)
2025b)
(Scaled Preference
Multi-Attribute
Paper
Wang
al.
Worse than ran- dom (93.0 + 92.9)
2025b)
(Pairwise Preference
Multi- Autti-Alttibute)
SMORM
Mutual improve-
(Pairwise
+ Multi-
ment of both heads
Attribute)
Specifically:
WWag al. 2025b) does not explore co-training. Their conclusions limited to initializing BT models with regression weights in specific datasets.
Even in that case, when using pairwise preferences (e.g., chosen/rejected labels), perfor- mance decreases after initializationnsuggssiiin instability or incompatibility.
Our finding is novel: through joint co-training pairwise preferences (BT head) and multi-attribute regression labels (regression head), we observe mutual benefits during training. These results indicate the two heads help each other learn better representations.
(3) Generalization Beyond Dataset Constraints. In Wang et al. 2025a). both reward heads trained on the same prompt-response pairs. In contrast, SMORM framework is explicitly designed to remove this constraint. That is, the BT and regression heads can be trained on different prompt-response distributions, generalization supported both:
Theoretically by formal analysis in Sec.
Empirically with experiments in Sec. 5 that demonstrate robust performance even when the heads are trained on distinct datasets.
18

---

Published as a conference paper at ICLR 2026
Moreover, (Wang al. 2025b) only considers single attribute (e.g.. Helpfulness) in regression. We extend this to a true multi-attribute regression setting, both in theory and in practice, making SMORM applicable in more realistic, uttribute-rich environments.
B SIGNIFICANCE OF OOD CHALLENGE
In our paper, we define prompt-response pair distribution shift as an out-of-distribution (OOD) setting, where the prompt-response pairs used to train the reward model and those encountered during online RLHF are drawn from different datasets. Prior works on reward hacking typically assume an in-distribution setting, where both the reward model training data and the policy optimization data (e.g., for PPO) are sampled from the same distribution.
Furthermore, this shift arises not only from novel prompts. but also from responses that evolve the policy is optimized. After just a few PPO updates, the model may begin to generate responses that move beyond the distribution of the offline preference data, as shown (Dai et al. 2025).
Moreover, while the base LLM benefits from broad coverage due to Internet-scale pretraining, the reward head is typically fine-tuned on much narrower preference dataset. As result, the inductive bias of the reward model is governed by this limited supervision, leaving significant room for generalization error when the policy encounters unfamiliar regions of the output space.
Although datasets like UltraFeedback contain over 250K prompts, training a reward model on the full corpus is computationally expensive. Our preliminary results (see Figure 2 show that even when the reward model is trained on high-quality subset such as Mkywork-80K-wwal which achieves strong pairwise comparison performance- the model still exhibits OOD vulnerabilitics. This illustrates fundamental trade-off:: smaller high-qualit, datasets offer efficient training but limited coverage, making them susceptible to OOD issues. Therefore, under the realistic and common scenario of limited training data and compute resources, OOD challenge is both prevalent and critical address.
C ADDITIONAL EXPERIMENTS
C.I TRAINING CURVES
In this section, we present experiments comparing our SMORM model with the GRM framework and the baseline multi-objective regression model, focusing on both training dynamics and evaluation performance.
Specifically, we fine-tune the base model meta- -.18----------tttttrteee using datasets: the preference dataset Skywork/SkyworrkReeee Pww.warddPrefeeeneeeeee dataset nvidia/HellpSeerr.
and the multi-objective reward
For comparisons against GRM, we report evaluation results on 10K samples drawn from as well as 10K out-of-distribution samples from
Skywork/Skywork Sreeward-reffeeece--000 we-bllnnde/Unniffee UnedieeeFFeedback to assess robustness.
For comparisons against baseline multi-objective reward models, we evaluate on both the training and validation splits of nvidia/HelpSteer2. We report metrics on the five granular attributes heipfulnsss,,,, correctness, coherence, complexity, and verbosity-using both mean squared error (MSE) and pairwise preference accuracy. To construct pairwise validation set, we take pairs of responses for the same prompt and derive ground- preference labels for each attribute by directly comparing their human-annotated attribute scores.
The results comparing SMORM-F to GRM are shown in Fig. From the figure, we observe the following: (1) As shown in subfigure (a), although SMORM converges more slowly than GRM during the early training stage-due to the need to jointly optimize both the Bradlcy-Terry loss and the regression loss-its training loss drops sharply after around 100 steps and eventually becomes lower than that of GRM. This indicates that incorporating the regression loss helps the BT loss converge more effectively. (2) Subfigure (b) shows that SMORM-F attains comparable in-domain performance relative to GRM. (3) In contrast, subfigure (c) illustrates that SMORM achieves substantially better out-of-distributtio performance-impoonn improving by approximately 20 points the UnifiedFeedback
19

---

Published as a conference paper at ICLR 2026
dataset. This demonstrates that SMORM notably enhances the generalizability of the trained reward model.
The results comparing SMORM to the baseline multi-objective reward model are shown in Fig. 7
(training MSE curves), Fig. S (training pairwise preference accuracy), Fig. (evaluation MSE), and
Fig. T0 (evaluation pairwise preference accuracy). From these results, observe the following: (1) As shown in both Fig. 7 and Fig.
9 although SMORM exhibits a higher MSE at the early stage of training, its MSE decreases sharply and eventually becomes comparable that of the baseline model. (2) For pairwise preference accuracy, SMORM consistently outperforms the baseline model, with the improvement being especially pronounced on the evaluation set, as illustrated Fig. 10
Meanwhile, to better explain how SMORM enables BT training to enhance the performance of the regression head, we analyze the correlation between attribute-level ground-truth preferences and the pairwise preferences predicted by strong single-objective reward model. Specifically, we use Skywork/Skywond -eword- V2--ama--- 1-8B, a model with strong scoring capability, to score all pairwise comparisons in the nvidia/HelpSteer2 dataset. We then compute, for each attribute, the proportion of pairs whose preference direction matches the model's predicted preference. The results are shown in Table 4
We observe that helpfulness, correctness, and coherence exhibit substantially higher correlation with the single-objective reward model, indicating that these attributes align more closely with well- trained Braddeyy--eoee -erry signal. Consistent with this, Fig. shows that the performance improvement of SMORM over the baselines is also most pronounced these three attributes, which supports hypothesis that BT training provides stronger learning signals for attributes that align well with the underlying preference structure.
Table 4: Alignment statistics and correlation percentages across different attributes.
Attribute
Alignment
Misalignment
Ties
Correlation (%)
helpfulness
5434
1787
2941
75.25
correctness
5149
1646
3367
75.78
coherence
3062
1143
5957
72.82
complexity
1696
970
7496
63.62
verbosity
3048
2033
5081
59.99
1.50
1.0-
GBM
SHORM
8T Loxs
1.00
Accuracy
Accurary
0.8
0.6
0.74
GRM
GRN SMORM
SMORM
0.4
100
500
Step
Global Step
Global Step
Figure 6: Comparison of SMORM and GRM on training dynamics and generalization.
Helph.tree
Correctress
Coherence
Camplaxty
Verbosity
socelie 2400H
2.5
Nonine
2.0
3.0
Shep
Stopp
Step
Shep
Step
Figure 7: Training curves on MSE.
20

---

Published as a conference paper at ICLR 2026
HMB B95S
Cineesse
Ourgeosty
Boodi: MM000
0.35
0.53
C20
0.90
0.99
01854
8.55
0.0
2400H
0.92
202
Figure 8: Pairwise preference accuracy on training dataset.
Phelctunnss
Correctans
Coherrence
Complesity
Verbousty
SMORM
SPOM
SMORM
200
670
653
200
400
130
200
200
600
200
800
200
200
500
Figure 9: Evaluation curves MSE.
C.2
ERROR BAR
To demonstrate the statistical significance of our method compared with the baselines, we evaluate
SMORM against GRM (Yang et al. 2024) and ODIN (Chen et al. 2024) through two sets of experiments:
(1) We use google/gemma- 2etpll/gemma------ii it as the base model and train SMORM on two datasets: the preference dataset Skywork/Skywork- Beeelyoooo--Rewrr-PPee Seerrd-Pfffreeee-800 80K-v0.2 and the multi-objective reward dataset nvidia/HelpSteer2. We run each method with three different seeds (42, 123, 456), and report the mean and variance across (i) training and evaluation curves on both the in-domain (Skywork) and out- of-domain (UnifieFFeedback) datasets, as shown in Fig. and (ii) performance on RewardBench, as shown in T
(2) Following the experimental setup in Sec. we again use gemma-2b-it it as the base model and train with three sceds (42, 123, 456), reporting the mean and variance of PPO-based RLHF experiments. The corresponding results are presented in Fig. 12
Across both experimental settings, SMORM consistently shows higher performance compared to ODIN and GRM, demonstrating statistically significant improvements.
We should also include results on RewardBench
Table 5: Mean + variance on Rewardbench.
Method
Chat
Chat Hard
Safety
Reasoning
Overall
SMORM-F
89.1 1.96
65.6 0.04
77.65 3.42
71.3 + 0.36
75.9 + 0.44 74.0 + 0.36
GRM
86.05 + 0.06
66.0 0.16
74.2 + 0.09
69.85 + 1.32
ODIN
84.25 2.40
63.05 1 0.12
71.75 + 0.42
73.75 + 17.22
73.2 + 1.44
Catsdrrop
Drsplarty
Coleoryy
0.53
SOORM
Eeseline
0.50
Dandine
0.70
SHOW
1.64
0.99
B525
Bruddin
640
0.29
0,45
0.43
0.00
330
400
CCB
300
650
200
480
800
200
430
500
200
403
300
Figure 10: Pairwise preference accuracy evaluation dataset.
21

---

Published as a conference paper at ICLR 2026
Skywork
Unified
0.70
0.95
0.65
Accuracy
0.90
Accuracy
0.60
0.85
0.55
SMORM
0.50
SMORM
ODIN
0.70
0.45
200
400
600
200
400
600
Step
Step
Figure 11: Training curves with error bar.
SMORM
1.4
SMORM
ODIN
ODIN
GRM
1.2
GRM
Proxy Reward
Golden Reward
1.0
0.4
0.6
0.0
10000
20000
30000
40000
50000
60000
10000
20000
30000
40000
30000
Training Samples
Training Samples
Figure 12: PPO curves with error bar.
C.3 RESULTS ON DIFFERENT OOD SETTINGS
To further demonstrate the the experimental setup in Hong et al. (2025) and compare SMORM (regression head trained on the multi-objective dataset avidia/HelpSteer) with the baseline BT classifier and its variants: BT-Norm (Wei et al. 2022), BT Bin-Hinge, BT-DR (Yuan et al. 2024). and BT-BSR (Hong et al., (2025). The results presented in Table 6 We observe that while SMORM achieves performance comparable the strongest baseline, BT-BSR, on the in-domain (ID) setting, it consistently outperforms all baselines across the three OOD evaluation settings, demonstrating its superior generalization capability.
(OOD) generalization ability of our SMORM, we follow
Table Performance of Llama-3 (3B) and Qwen2.5 (3B) under different disjoint evaluation settings (values multiplied by 100).
Setting
Model
LBt
LBT-Hinge
LBT-Norm
LBT-DR LBT-BSR
SMORM-F
Llama-3 (3B)
84.5
84,0
80.0
82.5
85.0
85.5
In-Domain
Qwen2.5 (3B)
84.5
84,0
82.0
84.5
85.5
87.0
Llama-3 (3B)
68.0
68.5
60.0
67.5
70.5
73.5
Prompt-disjoint
5wen2.5 (3B)
70.0
69.5
65.0
67.5
72.0
74.0
Llama-3 (3B)
61.5
61.5
50.0
58.5
63.5
68.0
Response-disjoint
Qwen2.5 (3B)
58.0
56.5
45.0
54.0
60.5
66.5
Llama-3 (3B)
51.5
52.0
47.5
50.5
54.5
57.5
Mutual-disjoint
Qwen2.5 (3B)
58.0
56.0
53.0
55.0
61.0
66.0
22

---

Published as a conference paper at ICLR 2026
10.0
7.5
5.0
Mean Score
2.5
-5.0
Baseline SMORM
-7.5
160
200
300 Step
400
500
600
Figure 13: Caption
C.4 MEAN AND VARIANCE OF SINGLE-OBJECTIVE SCORE
In this section, we report the mean and variance of the single-objective (preference) score during training. Specifically, we follow the experimental setup described in Sec. C.2 and plot the preference scores provided by SMORM-F and the baseline BT classifier. The results are shown in Fig. 13
We observe that the mean score of SMORM-F exhibits a noticeable shift at the beginning of training and stabilizes after approximately 10 steps. This trend aligns with the results in Fig. and Fig. 9 where the MSE of the regression head rapidly decreases during the first 100 training steps. This observation empirically supports our theoretical result in Prop. [ which establishes connection between the BT scores and the regression scores trained jointly via SMORM.
C.5 PRELIMINARY EXPERIMENTS ON IMPROVING BOTH SINGLE- AND MULTI-OBJECTIC HEAD
Experimental Setup. We follow the same training setup for SMORM described in Section 3 We compare SMORM with the baseline SORM and MORM, using Llama-3.22 3.---33.----------- ruct the backbone model for all methods. The comparison results RewardBench against SORM respectively. The corresponding datasets used for training each reward model are also listed in the tables. Note that SORM is trained solely on Ds. MORM on M,, while SMORM is jointly trained on both Ds and D
and MORM are presented in Table and Table and
Table 7: Results on RewardBench compared to the baseline SORM.
Dataset D3 / DM
Model
Chat
Chat Hard
Safety
Reasoning
RewardBench
UlrrFecedbakk (binarized)
Baseline
89.1
40.7
45.2
36.7
52.9
SMORM-M
91.3
39.0
50.1
41.9
55.6
UlraFeedback
SMORM-F
88.4
43.4
44.3
49.6
56.4
Baseline
73.4
60.5
79.8
49.6
65.8
Skywork80K/
SMORM-M
83.5
55.5
74.5
53.6
66.8
HelpSteer2
SMORM-F
80.4
62.1
80.7
55.1
69.6
Results Analysis. From Table we observe that SMORM-F achieves a higher average score on RewardBench compared to the baseline model. Notably, regardless of whether the single-objective reward model is trained on the same dataset or a different one, SMORM-F consistently outperforms the baseline. This result highlights the flexibility of our SMORM framework. From Table further observe that by simply aggregating the multi-atrribute scores to create a single-objective preference dataset and training SMORM. SMORM-L achieves average scores 5.5 and 6.3 points higher than training a MORM alone. This highlights how easily SMORM can enhance the performance of the multi-objective reward function.
23

---

Published as a conference paper at ICLR 2026
Table 8: Results on RewardBench and RM-Bench compared to the baseline MORM.
Datasct Dş DD
Model
Chat
Chat Hard
Sufcty
Reasoning
RewardBench
RM-Bench
Baseline
55.8
50.4
44.8
54.2
51.3
49.2
HelpSteer2 (binarizedy
SMORM-M
48.9
48.5
51.4
75.7
56.1
50.1
HelpSteer2
SMORM-F
50.3
48.7
54.7
73.6
56.9
50.9
SMORM-L
50.7
49.3
52.9
73.3
56.8
53.0
Baseline
70.3
46.1
41.6
42.0
50.0
50.2
UltraFeedback (binarized)/ UltraFedbacc
SMORM-M
91.3
39.0
50.1
41.9
55.6
51.0
SMORM-F
88.4
43,4
44.3
49.6
56.4
49.8
SMORM-L.
90.2
40.1
54.2
40.8
56.3
54.4
C.6 ADDITIONAL RESULTS ON IMPROVING SINGLE-OBIECTIVE HEAD
In this section, we provide additional experimental results comparing SMORM-F to baseline single- objective reward functions. We follow the same experimental setup as in Section 5 Mistral-7B-IIntruee as the base model are reported Table
Results using
Table 9: Comparison SMORM-F and baselines on RewardBench. Baseline results from Yang et al. 2024).
Ds/Dari UniftedFeedback 400k/UtttaFeebbuck
Ds/Day UnifiedFeedback 40k/HelpSteer2
Reward model
Chat Chat-Hard
Safety
Reasoning Base Model: Mistral 7b Instruct
Avg
Chat
Chat-Hard
Safety
Reasoning
Avg
Baseline (Single)
96.6
52,4
86.7
69.5
76.3
94.9
51.7
64.9
62.4
68.5
Baseline margin
96.4
51.5
64.8
74.5
89.7
47.1
70.7
43.6
62.8
Label smooth
97.2
49.8
85.8
72.3
76.3
94.1
47.1
67.5
79.7
72.1
Ensemble
96.6
51.8
73.0
76.6
89.6
50.2
59.0
69.3
GRM (lineur) w/ dpo
98.0
53.3
86.4
75.3
78.3
95.1
47.5
74.7
74.9
GRM (lincar) w/ sft
97.8
54.6
86.3
79.2
79.5
93.4
51.9
80.7
78.8
76.2
dpo
74.4
78.0
52.4
77.3
76.4
GRM w/ sit
98.0
55.3
85.8
71.2
77.6
94.1
48.5
83.4
77.4
75.9
SMORM-F
97.8
55.3
85.9
80.1
79.8
95.8
60.1
80.5
74.7
77.8
D THEORY
D.1 DEFINITIONS AND ASSUMPTIONS
Let K be the number attributes considered. We define fo as the backbone of our reward model, which maps input question- response pair to a hidden representation of dimension d. For each attribute head k, the reward score is given by гı w fo. In particular, тs (or ro) denotes the score from the single-objective head used to model overall preference (e.g., chosen rejected), while rk for & > 0 corresponds to the outputs of fine-grained attribute-specific heads.
Positive-definite covariances. Let
fe = (x., fr = fo(oovo) fn fo(mm Vm).
We define the covariance matrices
Σs = EDs [ - Jr) [f) [ and assume both are positive definite (PD).
and EM = Eos [mmfm]]
Modern feature extractors (e.g., transformer- backbones) typically embed inputs into high- dimensional space of size d, which exceeds the intrinsic rank of the data. Provided that the samples D: (or D M) do not all lie in a strict lower-dimensional hyperplane, the corresponding empirical covariance matrices will be full-rank and therefore positive definite.
Correlation between heads. naturally assume positive correlation between the aggregated fine-grained attribute scores and the preference along the chosen/rejected dimension. Specifically, let
μ1 M>D [So ve) fo yr)]
24

---

Published as a conference paper at ICLR 2026
and define
Cm ==
[fo Fn rr]] E R
Note that we denote the reward function, and the vector of multi-attribute scores. Then, define
us 2M E. Cm:
and assume that the sum of its components non-negative, i.e.,
1 a O.
Assuming E(r) = [ = 0, we define the raw coupling vector
Ms MM CM
Cov(ri, wff),.... Coo((,, w3f] E RK
and normalize
Ps M Cas E TR=
a
B
Ass
Ug Eg us
Then, 3; if the score attribute i tends to increase the single-objective score w increases.
The sum 1 represents the total (signed) covariance between the single-objective preference score and the aggregate multi-attribute score. In most real-world annotation settings, chosen-rejecter labels used proxies for overall answer quality. Since higher-quality responses tend to improve multiple fine-graid graiirr attributes (e.g.. helpfulness, correctness, coherence, etc.), the covariances across these attributes typically sum to a positive total.
Hidden Ground Truth. For each head rk, we denote rī as the hidden ground-truth reward function, and we model the labeled score for head k as 9k r + ek for NO, E), Eox > Here the diagonal entries of Σ are σbk Var(En), the noise variance for head k.
D.2 PROOF OF THEOREM 1
Theorem 1 (Implcit Multi Attribbt Effect), Let a reward model be trained under the SMORM
framework, and suppose the following conditions hold: (1) Bounded features: There exists
B oc such that [) 3) S B for every (x, y)- (2) Positive-defiite covariances: let fo = So( J and Σm := -EDD f fml are positive-defiitte matrices. (3) Positive correlation: Let us == B(E.V....))DD (fo(x.. fo (())) and let Css ;= ((Em..mm)))DD Ifo (I Vm,) r Then a M Ca has nog-tigative sum, i.e. 1Ta ward heads converge to their population minimizers, there exist constants
fr. Lo(#:: vr). fm = (m)) Eg == EDs((S [r)
Raxk
O. As the optimization of both re-
1Ta
and
[p] Eg *ss)
2 0 depending only on B and second-order moments- such that for every pair (2,y), y): w
(m( I Jo 2 c cww fola,, ())
(r, E.
(6)
Replacing the logistic (BT) and regression losses by squared losses does not alter directions of the minimisers because any strictly convex proper surrogate has the same first-order optimality conditions up to positive scalar factor Bartlett et al. (2006). We therefore analyse the following least-squares problems:
Proof.
min [s[(w] (Jc fr) 1)
min EM || WT -r1
For a generic least-squares objective minw B[(wTu f)3], setting the gradient to zero yields Eluu u w E[[u i]. Applying this template we obtain the population solutions:
ws = E5
-ー1 Hs:
WM = ST
Define the whitening operator 1 == Eg 1/2 E Rd, As = 1/2 us. and Čas == E1 E Ca-
1/2
Then C.
Σ5 1/1 fer Dr = 1-1/2 fr and Es(( - ( - 2,) [| = la and jas + Then
Equation becomes:
Ws Eg -1/2 1/2
WM = Eg 1/2 1/2CMM
uss.
(8)
25

---

Published as a conference paper at ICLR 2026
Because jss + O, we write the Euclidean projection of each column of ČM onto As:
ig CMCM
ÖM
us
+E,
E ns 0.
(9)
L
aligned component
Define the coupling vector B = and thus 1 3 O.
2 C.CMM M
Mas E R.K. By assumption (3) we have 1 aT 2
Then for feature vector S, we have:
Equation B
T Σg 1/2
1
W M
Equation ( ④) E +
3WWW5f) E Φ
Decomposing by:
Pg |
As +
Ug = 0,
Lis
llAs
so that E ④ ET z. By bounded-features, ||
B and hence || S B/ Amin (Ms), where / Ami [ms) is the square root of the smallest eigenvalue of the The projection term has norm Kig / /llll < [w] fl. so the orthogonal part satisfies || < B/ Amin (Es). Therefore
B
L 2|
|Bllop |||||
oop
Amin Es)
Averaging the K attributes and using max(0, 10 gives
K K
Whe 2 cw3s 8
(10)
K
where c a K
K
Monotone lower bound. Because 1 T 0 by assumption (3), we have 0. Define the linear function L(s) = cs E/K. Inequality equation 10 reads
(x,
Σ TT.. (M.oo(
I(rs(x,)))
TM is bounded from below by the increasing map L(.) ) of the single-objective score rs. Conse- quently, for any two responses (#1, 3/1) and (z', (Z),
(s(m1,91) 2 rs(#2, y2)
((s((1,81)) > Urs(#2, v2)),
and hence the lower bound on the multi-attribtee average larger (or equal) whenever the single- objective score is larger. In particular, for any threshold т,
rs(x,y) v)
(ar(2,)) 2 CT K
This completes the proof of Theorem T
Pure-SORM failure. If no multi-attribute head is trained (CMs 0). then = 0 0, so the lower bound degenerates tor 2 -E, offering no positive coupling between single-objective and multi-attribute scores.
26

---

Published as a conference paper at ICLR 2026
D.3 PROOF OF LEMMA
Lemma | Let y yn be pair of responses. Assume Is (y) is the ground truth score and v's (y) is the predicted score under Brudley-Terry model. Then:
P(yA + yB) = a (r=(ua) rs(un)), P*(yA + yB) a (gs(va) gs(u)))
where (t)) Ite The expected preference enror satisfies:
EDs [PYA > yB) PM PP UB)l BDD (V2MSE(T.))
with MSE(rs)) = (r.(v) Io (v)) Similarly, for a multi-objective reward model with predicted score I' and ground truth gm. let: cm I' (yA) Tm Ty(B)), e = Im (ya) 9m (yB), then the error is bounded as:
EDM 1em -m 5mls I EDN (V2MSE(Fm))
Proof. For a pair of responses and yn, the Bradley-Terry model defines the probability that уı is preferred (i.e., has higher overall reward) as:
P(UA + yB) (ra(vA) ruB)),
where (t) Ite is the sigmoid function, and r, (v) is the model's predicted overall score. The corresponding ground-truth preference probability, based labeled scores, is given by:
P (yA > vB) = (9.(HA) - (=()))
where 9-(-) denotes the ground-truie truth reward.
The prediction error in probability space is the absolute difference:
AB (ra(AA)) r.()))) σ (96(2A) (o(U)))
Since the sigmoid derivative satisfies
= (1)= - (b))
and reaches its maximum value of 1 at 0, have o'(t) < for all t. This implies that the sigmoid function is 1 L-iisschitz:
(o(a) o
1 la b). Va
Applying this to our setup, we obtain:
1 4 (rr((v)) rr(BB)) (ua( 9, (UB))
P(yA yB) Pp* (VA > UB)l S
Taking expectation and applying the Maucch-Schwarzee inequality, further bound the expected pairwise error by:
Pairwise-error < V MSE(r.),
where MSE(r.) = E,, [(r.(y) (v))² is the mean squared error of the predicted scores.
This result shows that minimizing the pointwise MSE of the reward model also reduces the upper bound on the pairwise misordering error, thereby improving preference consistency.
Similarly, for multi-objective reward model with predicted score I'm and ground truth Jm, let: en Tm (VA) - Tm (UB), ex - -9m(wA) - (VD),), then the error is bounded as:
EDY Iem eial EDA ( (V2 (22MSE(r.))
27

---

Published as a conference paper at ICLR 2026
To relate this to the Bradley-Terry loss, recall that the Bradley-Terr loss for pair (yA, UB). where VA is the preferred response, is given by:
BT log (r. (VA) re (UB)).
If model's predicted scores r. (v) are close to the ground-truth scores 9s (x(#) (v)-ie.. the mean squared error (MSE) small- the difference (yA) -r,(wB) will elosely approximate (=(ys) Is (yB) By the Lipschitz continuity of the sigmoid function, this implies that the predicted probability under the model and the ideal ground-truth probability will also be close. Consequently, the pairwise preference error will be small.
This reasoning provides theoretical justification that minimizing the MSE of individual predictions naturally leads to accurate pairwise probability estimates, evaluated by the Bradley-Terry loss.
Furthermore, while the single-objective head r, is trained using the Bradley-Terry likelihood, our previous bound shows that the expected Bradley-Terry test risk.
RBT E(yA.9)) [- log ( (VA(AA) - rss (un))]
is a I-Lipschitz function of the score difference r, (UA) - (UB).. Therefore, controlling the vari- ance of the individual scores- capture by MSE, = E[(r.(v) ( bounds the generalization error under the Bradley-Terry loss, up to constant factor.
D.4 PROOF OF THEOREM 2
Theorem 2 Under the same assumptions as in Theorem | and assuming that the feature extractor fo is differentiable, let f denote the maximum likelihood estimator (MLE) of the ground truth optimal parameter 8". Let 0, and 6m denote the maximum likelihood estimators of the single- and multi- objective reward functionn, respectively. Define Ms(y) = w fo- (y), My) = w fo- (у). Then, for a response y, the mean squared error (MSE) of the predicted reward can be approximated
MSEş R (oMs Cov (0.) VoMs(u) + doo, MSEM R3 VoMaa(y) Cov (Bm) VoMa(yy) + 0oo.
where Too is the intrinsic randomness in the label. Moreover, SMORM yields lower asymptotic MSE for both the single- and objtivobjective heads compared to training either head alone:
MSE
< <SS
MSE
< MSE
(11)
Proof. Fisher matrix. The Fisher information is a way of measuring the amount of information that an observable random variable carries about unknown parameter. Mathematically, for parameter vector 0 and data D with likelihood p( D 0), the Fisher information is
I(0) def
pD 10)
V P(D 0)V, log p(D []
(12)
Intuitively, measures how sensitive the log-likelihood is to small changes in 0. More curvature means larger [0) and thus implies that can estimate more precisely. The celebrated Cramér-Ra Rao
bound says that (under mild conditions) any unbiased estimator's covariance is least [I(0)] Kay 1993).
In our square-loss, Gaussian-nove noise setting, the empirical Fisher matrix becomes the empirical sum of outer products of gradients:
I (rogimo)) (0)
Σ 7
Σ [oraaa
TRk
i=1
For single-head reward model that evaluates the overall quality, have K. (0). For K-attribute reward model that evaluates the response according to K specific aspects, K- (I..., K}. For our hybrid model, K {O.... ......KK K}. Because every summand is positive semi-definite. adding task can only increase or keep the Fisher matrix. Hence we have:
(byyrid))
(single) A
A x 0,
(13)
28

---

Published as a conference paper at ICLR 2026
Striet positivity of difference term. In fact, give the assumed positive correlation between head 0 and other attribute heads, we can show that the overall Fisher matrix can be strictly larger. Denote
go (v:) = (Jis
which is the gradient of the overall head's prediction with respect 0. When we look at the contribution of the other heads, what matters is how their gradients project onto go (vi):
go(u)) (or)(
The positive correlation assumption Pot > 0 implies that, on average, the gradients (orx(us) tend
to point in a similar direction to (o()).. This means that go Verk (3)) > 0. For example, project the Fisher information onto the direction go, using the linearity of the inner product, obtain:
K
(o(y:) h (0) (3:))) = (3())) (T Izaingk (0) 90 (v:)
RO Σ (90(v)) Vork (ork
Because each term (3) (orr(y))) strictly positive when the inner product nonzero, and positive correlation ensures that it is indeed positive on average, we know the extra sum is strictly positive. That
(y:) T hyybri (0) (v(()) > (1,) s (0) gu()))
Therefore, we can get
(14) Asymptotic Variance of O. If 0 is the maximum maxinum-likelihood estimator (MLE) of 0* and the usual regularity conditions hold (i.id. samples, smooth bog-likelihood. finite Fisher information, etc.), then the asymptotic normality theorem for MLEs states
(byyridd = (single) + +A,
~
vπ ( 8*) S ( ((00
where " "A denotes convergence distribution and the covariance of limiting Gaussian the inverse Fisher information, which is the smallest possible asymptotic variance for any unbiased estimator by Cramér-Ras Rao.
Thus, if one yields a larger Fisher matrix (more information), its estimator's asymptotic covariance matrix is smaller, so predictions based on are less variable. Hence
Cov hybrid < Cov single(A).
(15)
From d-variance to MSE by Bias-variance decomposition. For fresh test example we predict with
$o w Ma(u),
gv(v) = w (v) + Eo.
The mean-squared error of that prediction is
MSF
(B(5o]
E [[g=])
Var[So]
Var(eo dootiieducible noise)
estimation variance
Bias term. With sufficient optimization and model capacity the MLE is (asympiotically) unbiased, this term is approximately
Variance term. Fluctuations of Ô across data sets propagate through the network, and first-order Taylor expansion gives
Mg(u) = fo (v) Volo (v) @ 0*)
-50(v) R (o) 0+) (denote Ms(v) def w (v)) Var voMs(u) Cov (or
Because the hybrid regime has the smaller Cov(@) by Equation 15 above, this variance shrinks.
29

---

Published as a conference paper at ICLR 2026
Noise term Ooo- This is the intrinsic randomness in the label and is identical for all training regimes.
Therefore, when hybrid training reduces the covariance Cov the key variance term in the general- ization bound decreases. Due to the assumed positive correlation between the fine- fine-grained attributes and overall quality, this reduction leads lower test-set MSE for the single-objective head. This establishes inequality 11 and thus completes the proof of Theorem
Moreover, by linking the single-head MSE the pairwise preference error shown Lemma T]. we demonstrate that the single-objective head trained using our SMORM framework is expected to outperform a conventional single-head reward model. It worth noting that the same argument naturally extends to the multi-objective reward setting as well.
Theorem 3 (Implicit Multi-Attribute Effect under Finite Data), Let reward model be trained under the SMORM framework. Assume:
1. Bounded features. There exists B < oo such that II (2; v) S B for all (xr, v).
2. Positive-definite covariances. Let fc fo (x..))))) ye). frr fo(Ix. fm = fo(Im 9m.) and EM ED [m and suppose both
Define Eg == EDg (-)(fc-fr) positive definite.
5r)
3. Positive correlation. Let ps B. Be) fo( y-)l and Cas E (xm: r DDr If (Im. ym r Raxk Set a == μ MM Cm and assume 1 2
Let pret and natt be the sample sizes used to train the BT head on Dg and the regression head on DM, respectively. Let wi and W as be the (empirical) minimizers obtained from these finite samples, and define
ra(2, v) := w/ fo(x,v)
Fo(z,)) = : 16 fo(x,
K
K
Tmn (x, = K
w MM,So(
(m = Σ Co,,SolI,
As the optimization errors vanish (each head reaches its empirical minimizer), there exists
aa
2 0
and 2
K (us E Us)
(depending only on B and second-order moments) such that, for every pair (x, y),
Fm, y) 2 cf,
n18
7/4
(16)
where the finite-sampe estimation errors satisfy
Op
Opl
Ms
Peprot
NM
=
Platte
Equivalently, replacing empirical heads by their population counterparts yields T' (x, y) cr.(x,y) and equation quantifies the conservative degradation under finite data.
Remark (what the rj's absorb). The terms 1)s and 7ι capture the statistical errors from estimating Jss, Es. EM, CM and the induced parameter errors in 0g. War- Concretely, one may bound them in terms of deviations like || psll. |اg Esll. || ΣvIl, and ||ट.r Car ||. which are O, (1/ppret) and (1/nnatt))) respectively, under the boundedness assumption and standard concentration.
30

---

Published as a conference paper at ICLR 2026
E ADDITIONAL CLARIFICATIONS ON THEORETICAL ASSUMPTIONS
E. RANK-DEFICIENT COVARIANCES AND THE VALIDITY OF THEOREM
In high-dimensional embedding spaces, the covariance matrices
Eg [E[[e= f+) f,)
Em Zm=Rffmf
may be singular. In this section we show that Theorem continues to hold under rank deficiency. Importantly, singularity affects only the magnitude of the coupling constants e and E; the implicit multi-attribute effect itself remains valid.
Least-Squares Solutions under Rank Deficiency In the proof of Theorem 1, the population minimizers of the BT and regression objectives satisfy
Egws = 148,
EMruM = Cm.
where μıs Blfe f+] and Cm E] When Eg or ΣM is singular, these linear systems may have infinitely many solutions. The correct interpretation is the minimum-norm least-squares solution, given by the
pseudoinverse:
Wy = -S
WM = -
The pseudoinverse coincides with the actual inverse in the posiive-definite case, and therefore generalizes the original proof without altering its algebraic structure.
Whitening with Singular Covariance
Define the whitened feature representation
11/2 J. === ES T = -=
t1/2 ps.
ČM = - 11 Cas-
The matrix // acts as an isometry on the row space of Es and maps all vectors the null space to zero. Thus applying 2s f1/ has the geometric interpretation of restricting the analysis to the identifiable subspuce-the subspace in which Eg has nonzero variance.
Within this identifiable subspace, the whitened covariance satisfies
Ei ( ②,) PIow:(E):
the orthogonal projector onto the row space of Es. This reduces to the identity when Es is full rank, and preserves the key orthogonality relationships used in the remainder of the proof.
Projection and Orthogonal Decomposition The next step in the proof decomposes ČM into the component aligned with ňs and the orthogonal residual:
ČM T + E,, E ias = 0.
This decomposition remains valid in the singular case because: (i) jos lies the row space of Es, (ii) C's also lies this row space, and (iii) the row space is Euclidean subspace where ordinary projection operations are well-defined. Therefore the projection step is unchanged: only the dimensionality of the subspace may be reduced.
Effeet of Rank Deficiency on the Coupling Constants
Within the identifiable subspace, we have
Th = (w] f) + TT
and by averaging over the K attributes we obtain the coupling inequality
Fm (x, > cr,
The constants and remain well-defined when pseudoinverses are used. However, rank deficiency can affect their values the following ways:
If Eg or EM collapses onto a low-dimensional subspace, then the aligned component
ig C' decreases, which reduces
The orthogonal residual E. may increase because fewer directions are available for alignment.
This enlarges the slack term
No matter the rank, the inequality preserves its form. Rank deficiency does not create inconsistencies; it only reduces the strength of the guaranteed coupling.
31

---

Published as a conference paper at ICLR 2026
Interpretation The situation mirrors classical linear regression with multicollinearity: singularity does not make the model invalid, but reduces the amount of signal that can be reliably extracted. In our setting, the implicit multi-attribute effect continues to hold; the guarantee simply becomes weaker when the embedding space contains fewer identifiable directions.
E.2 POSITIVE-DEFNNITE COVARTANCES
Our theoretical analysis does fundamentally rely on Es or Eas being strictly positive definite. When these matrices are singular, the population solutions in Theorem are interpreted using the Ponre- pseudoinverse:
Wg
Elgss,
WM Ele
and the whitening operator becomes 1/1// Here 2i and Mi, denote the Moore-Penrose pseudoin- verses. For any possibly singular matrix A, its pseudoinverse At is the unique matrix satisfying
AAAA= A'AA = A", (AAt) = AAT, (A'A) AAA.
It coincides with the usual inverse when A is full rank and otherwise computes the inverse on the identifiable subspace while nulling the kernel. Under this generalization, all algebraic steps of the proof continue to hold; only the coupling strength may decrease.
For convenience, we rewrite the relevant quantities Theorem in a fully explicit and self-contained way. Recall that the BT population solution is
ws Σg' uss
Ms = (I Me) f (x y-)]
and the multi-attribute regression solution is
WM En
CMr = B[f Bm) rxl.
where CM.K denotes the k-th column of CM-
Following the whitening step used in the proof of Theorem define
ïs = 1/2
1/2 Ps.
ČM,K : Σク1/25. CM.
Both vectors now live in the same whitened feature space where El(ie (f f.)
The coupling coefficient from Theorem 1 is
U. CAS
Bx
A
To decompose this expression, first introduce the normalized directions
ČM.K
|sll
[[C
Using the identity i.g C..K = ||As M I 6e), we obtain the factorization
||Carell
BK (As, tx)
||
Finally, the norms of the whitened vectors can be expressed in terms of the spectrum of Es. Let
Amax (Es)
Auan (Es)
denote its condition number. Since || E [|lpsI// / Amax. ||ssl// h Amian, the ratio [ kll | contributes an additional factor proportional to x-- 1/2, Combining terms yields the decomposition:
|'mall Assll
(fts, U1)
1/2
directional aligament
conditioning factor
signal-strength
32

---

Published as a conference paper at ICLR 2026
This expression makes explicit how singularity poor conditioning of Es affects the coupling: if Es becomes low-rank, then (i) the projection CM.& onto the identifiable subspace shrinks, reducing (As, 0x); (ii) the magnitude |[MM, decreases because components in the null space are removed; and (iii) x- 1/2 decreases as the spectrum collapses. Consequently, S5. (and thus c) becomes smaller, while the orthogonal residual in Theorem becomes larger.
Empirically, SMORM does not enter such degenerate regimes. The regression head achieves the same MSE as the single-head baseline, indicating that the attribute signals are not compressed into a low-dimensional null space. Likewise, the BT scores exhibit stable, non-collapsing variance (Fig. 13), implying that Es maintains a well-spread spectrum and that 6 remains O(1). Together these observations guarantee that stays non-trivial and remains small in practice.
We will clarify in the revision that positive-definttenss is used only to simplify notation, and that the correct formulation uses pscudoinverses and the identifiable subspace.
Response. We thank the reviewer for highlighting the issue of singular covariance matrices. Im- portantly, our proof does not rely strict positive definiteness of Σg Σm- When these matrices are rank-deficient, the population least-squares solutions in Theorem 1 are interpreted using the Moore-Penrose pseudoinverse:
WM = Ely CM-
With this substitution, every algebraic step in the proof continues to hold. The only thing that changes is the magnitude of the coupling parameters.
Intuitively, if the shared embedding collapses onto a liver-dimensional subspace, then fewer feature directions remain identifiable. In this case the aligned component of the multi-attribute signal becomes smaller, which reduces the coupling constant c, while the orthogonal residual may increase, which enlarges the slack e. The core inequality
I'm vx 2 cr,( v)
remains valid, but with potentially weaker and larger e in highly degenerate regimes.
Empirically, however, this pathological situation does not arise in the SMORM training regime. (i) The regression MSE of SMORM matches the single-head regression baseline, indicating that the multi-attribute directions are not lost low-rank subspace. (ii) The BT head maintains stable, non-collapsing variance (Fig. 13), implying that the effective Eg is well-conditie cellicondito and does not approach singularity. Together these observations ensure that the effective covariances retain sufficient rank that e non-trivial and e remains small in practice.
We will clarify in the revision that Theorem should be interpreted in terms of pseudoinverses and the identifiable subspace, and that positive-deffniteensss is used only to simplify notation, not structural requirement.
E.3 FISHER INFORMATION MATRIX
In large, over- wee-parameterized LLMs the *full* Fisher Information Matrix (FIM) is indeed singular. Importantly, our argument p.24 does ** **not** require the global FIM to be invertible. What matters for the asymptotic variance of the reward model is the Fisher information **restricted to the task-relevant subspace** not the entire parameter space.
##ffecttvee Fisher Information.** ** Even when the full FIM is singular, the gradients that actually affect the reward outputs,
9s ()) VyMs(y),
9M (v) VoMar(V))
span a low-dimensional #identifiable subspace. The Fisher information **rsttrccted to this sub- space*# is
( E[gsg5]
(M) E[gagT]]
ef
which remains positive definite long the reward functions not constant. This is the standard notion of "effective rank" used in modern statistics of deep networks: the identifiable subspace can have well-conditioned Fisher even when the full model has infinitely many uninformative directions.
33

---

Published as a conference paper at ICLR 2026
Accordingly, our asymptotic variance expression should be interpreted
Var[Ms(0)] ~ Us T(0*)* gs:
where )(0*)+ is the ** "MMorr-PPenrone pseudoinverse*** of the FIM restricted to the identifiable subspace. This is the standard formulation for asymptotic normality in singular (or non-identifiable) models.
((siggle) denote the effective Fisher W*y hybrid training helps even under singularity. ** Let eff information generated by the BT head alone. Adding the multi-attribute regression head introduces an additional positive semi-definite term
Eggggal = 0,
so that
(hybrid) ((single) A. lef off
This relation holds #rrgardeess of whether the full FIM is singular**, Since A 2 O, the hybrid model always increases the curvature in the wward-relevant directions and thus increases the amount of information available for estimating the reward parameters.
In the pseudoinverse formulation, this yields, along the BT direction gs+
(lybrid)+ 95 S 9s off
[(single)+
9s off
gs,
which means that the asymptotic variance of the single-objective reward functional Mg decreases under SMORM, even when the global FIM is singular. Intuitively. SMORM enriches the informative subspace where the reward gradients live, and thus stabilizes the estimator.
F MOTIVATION FOR ENHANCING MULTI-OBJECTIVE REWARD FUNCTIONS WITHOUT ADDITIONAL MULTI-ATTRIBES DATA
In this section, provide detailed discussion the limited availability of high-quality data for training multi-objective reward models, which motivates our approach to enhancing multi-objective reward modeling performance without relying additional multi-attribute annotations.
While several datasets provide dense prompt-response pairs with fine-grained attribute scores -ure UltraFeedback (Cui al. (023) with 240K samples and Prometheus (Kim al.) [223] with 200K samples-their annotations are primarily generated by GPT-based models. This introduces several concerns: (1) As foundation models continue to evolve, the quality and consistency of their annotations become increasingly difficult to guarantee.
(2) The use of large language models (LLMs) as annotators introduces potential biases (Gu et al. 2024 Li al. 2024), which can be inherited by the reward model and subsequently transferred to the policy model when optimized using reward signals. (3) Our experimental results in Section show that, when using gemma-26 as the base model, human-annotate dataset of 20K samples-yickds superior performance compared to training on UltraFeedback with 240K GPT-labeled samples.
training on HelpSteer2 (Wang et al.) 20241)
HelpSteer2 (Wang et al. 2024r) is one of the few available datasets that provide high-quality, human- annotated, oulti-objective preference labels. However, only contains 20K samples, and its creation involved exceptionally rigorous annotation process. This process includes multiple layers of human oversight, dynamic annotator recruitment, and strict quality control procedures to ensure data integrity. Due to the high resource demands of this pipeline, it is difficult generalize scale to larger datasets.
In summary, remains difficult obtain large-scale, high-quality, fine-grained attribute scores for responses, whether through GPT-based evaluation or human annotation. This limitation motivates the development of methods that can enhance the performance of multi-objective reward models without requiring additional annotated data.
G EXPERIMENTS ON OOD
To evaluate the generalizability of our SMORM, we adopt the experimental setup described in Section training reward models both 400K and 40K samples from the Unified-Feedhadd dataset.
34

---

Published as a conference paper at ICLR 2026
In both settings, HelpSteer2 serves the multi-objective dataset DM for SMORM. All methods use gemma-2B-it as the base model and are evaluated on both in-distribution (ID) data (Unified- Feedback) and out-of-distribttion (OOD) benchmarks (HHH-Alignment and MT-Bench. The results are presented in Tables T0 and [T] The results demonstrate that SMORM-F consistently outperforms competing methods in both in-distribution (II) and out-of-distribution (OOD) evaluations. Specif- ically, SMORM-F achieves an ID score of 76 and an OOD score of 83.2 on the HHH--lligmmet benchmark, surpassing the second-best method, which attains scores 73.8 and 79.6, respectively, These findings suggest that incorporating multi-objective learning function effectively shapes the embedding space, leading improved performance on ID data and enhanced generalizability of the single-objective head to OOD scenarios.
Table 10: Results on ID and OOD evaluation with 400K training data from Feefied-Feedback.. The best performance in each task is in bold and the second best one is underlined.
Reward Model
Unified Feedback
HHH Alignment
MT Bench
Classifier (Frozen)
63.8
66.4
69.5
Classifier (baseline)
72.1
73.4
71.2
Classifier margin
72.0
75.0
72.6
Classifier + label smooth
71.5
72.1
71.2
Classifier Ensemble
72.8
76.8
73.7
GRM
73.8
79.6
73.4
SMORM-F (Ours)
76.0
83.2
73.4
Table 11: Results on ID and OOD evaluation with 40K training data from best performance in each task bold and the second best one is underlined.
The
Reward Model
Unified Feedback
HHH Alignment
MT Bench
Classifier (Frozen)
63.9
68.6
68.2
Classifier (baseline)
68.8
70.3
69.1
Classifier + margin
69.6
69.8
71.0
Classifier label smooth
68.5
68.8
71.9
Classifier + Ensemble
69.9
72.2
71.1
GRM
71.3
75.1
72.8
SMORM-F (Ours)
72.9
77.4
73.0
H ALIGNMENT RESULTS AFTER PPO
To demonstrate the effectiveness of our SMORM in training superior policy models within the RLHF framework, we evaluate the win rate of models trained using PPO with SMORM-F as the proxy reward model, comparing it against two baselines: Baseline SORM and GRM. We obtain the optimized models following the training process in Sec. 5.3 and randomly sample 500 prompts from the Shified-Feedbace dataset and use the optimized policy models to generate corresponding responses. We then employ GPT-4o as an impartial judge to compare the model outputs and determine which response is preferable, recording the win rates accordingly. We adopt the win-tie evaluation prompt format introduced by (Khanov et al., 2024). The results are summarized in Table [2] The system prompt and user prompt used for evaluation provided below:
35

---

Published as a conference paper at ICLR 2026
Prompt for Win-Tie Evaluation by GPT-4o
System prompt
[system]
You are a helpful, harmless, and precise assistant for checking the quality the answer. We would like request your feedback the performance of two AI assistants in response the user question. Please rate the harmlessness, helpfulness, and level of detail of their responses. Your evaluation should consider factors such the helpfulness, harmlessness, relevance, accuracy, depth, creativity, and level detail of the response. Note that if a response appears off at the end due to length constraints, it should negatively impact the score. Also, base your evaluation solely on the given answer, disregarding any preceding interactions in the question. Each assistant receives overall score on scale of I to 10, where higher score indicates better overall performance.
Please first output single line containing only two values indicating the scores for Assistant 1 and 2, respectively. The two scores are separated by space. In the subsequent line, please provide a comprehensive explanation your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment. User prompt
[Question]
{question]
[The Start of Assistant 1's Answer] {answerl)
(The End of Assistant 1's Answer] (The Start of Assistant 2' s Answer] [answer2)
(The End of Assistant 2's Answer]
From the table, we observe that across both dataset scales, using our SMORM-F as the proxy reward function results in policy model that consistently outperforms both the Baseline and GRM, with win rates always exceeding 65%.
Table 12: Win rate of models after PPO training with SMORM-F against baseline SORM and GRM.
Method
VS.
Method
Win (%)
Tie (%)
Lose (%)
Ds/ :D:: UnifiedFeedback 40k/HelpSteer2
SMORM-F F SMORM-F
Baseline
75.7
0.5
23.8
GRM
69.5
0.4
30.1
Ds Ds/DM: DDM: UnifiedFeedback 400k/Ultraeeedback
SMORM-F SMORM-F
Baseline
71.3
0.7
28.0
GRM
65.7
0.7
33.6
I IMPLEMENTATION DETAILS
1.1 BASELINE AND TRAINING DETAILS
All baseline reward models use the catiiii class from the transf cransformrss rmers library (Wolf et al. 2020). which attaches a randomly initialized linear head for reward prediction. Each model is trained minimize loss function using the training data. For ensemble baselines, train three models with different random seeds and aggregate their predictions.
Baseline Details.
Aut www.ooeeeeeorreeeeeeeeeeeeeeeeee
We the margin loss from (Touvron al. 2023) defined as:
Emargin (0) --E( [log (a (ro(#, Mo) ro v.) m(r))))
where m(r) is computed using the reward difference between chosen and rejected responses in the Uni fied- Feedback dataset. This loss emphasizes meaningful reward distinctions.
36

---

Published as a conference paper at ICLR 2026
We also incorporate label smoothing loss defined as:
Lsmooth (0) 18 peww)))) [(1 c) log (a (re(a, me) ro(2, m=)) log (ra(2, ro(a, yr yr)))]
where 0.1. This formulation improves robustness by softening the loss against label noise, reducing overfitting.
2024). The reward head consists of linear layer (hidden size, 1024), followed by ReLU activation, and a final linear layer (1024, 1). The regularization coefficient is set to 0.01, and / is set to 0.1. the GRM-linear variant, the head is single linear layer of shape (hidden size, 1). More detailed training procedures for GRM can be found in Yang et al. 2024).
For GRM, we strictly follow the implementation in Yang
Computational Resources, All experiments were conducted using NVIDIA H100 80GB GPUs. Training on 40K data samples requires approximately 16 GPU hours.
Table 13: Key implementations of the text generation experiments.
Basic Information
Base models Quantization for training Optimizer
gemma- 2b-it and Mistral-7B- Distral-7BIInstrucc 1B-Instruct--0.2 bf16 Adam hf
Batch size
16 5 x 10
Learning rate Learning rate scheduler
cosine
Warmup ratio
0.03
SMORM
Weight ratio for single-objective to multi-objective reward modeling
1.0 by default
PPO Schulman et al. 2017
KL regularization
0.0
Epochs
Learning rate
x 10-5
for GAE
0.95
Clip range
0.2
Optimization epochs per hatch
Tokens during generation
512
J HYPERPARAMETER ANALYSIS AND INSTABILITY OF GRM
In this section, conduct experiments demonstrate the instability of GRM Yang et al. 2024 and the relative stability of our proposed SMORM. Intuitively, reward models are typically initialized from language models pretrained on next-token prediction tasks, and are then fine-tuned for reward modeling. This motivates the hypothesis that introducing a next-token prediction regularization term-alignee with the pretraining objective-mai conflict with the reward modeling objective. As a result, this misalignment could lead to unstable performance during reward model training. To validate this assumption, 40K samples from Uni modified-Feedbaae as Ds and HelpSteer2 as DM- We compare the performance of GRM In addition, we compare a baseline multi-objective reward model to our SMORM-L. We vary the weight ratio in [0.01, 0.1, 1, 10). In GRM, the weight ratio refers the strength of the next-token prediction regularization. In SMORM, the weight ratio controls the contribution of multi-objective reward modeling. All models are evaluated on Rewa Behhhch. The results are presented in Fig. [4] From the figure, we observe that, except in extreme cases, our SMORM framework demonstrates consistently stable performance. Specifically, when the weight ratio is set to 0.01, the performance of SMORM-L slightly falls short of the baseline multi-objective reward model, and when the ratio is set to 10, SMORM-F marginally underperforms compared to the baseline single-objective reward model. Outside of these edge cases, both SMORM-F and SMORM-L consistently outperform their respective baselines across a wide range of weight settings. In contrast, the performance of GRM fluctuates significantly, ranging from approximately 45 to 65, and consistently underperforms relative
conduct experiments following the setup described in Sec. [ using
baseline single-objective reward model and our proposed SMORM-F.
37

---

Published as a conference paper at ICLR 2026
to the baseline single-objective reward model. These results support our hypothesis that incorporating next-token prediction regularization introduces instability into reward model training. This instability is especially concerning given the substantial computational cost required to train reward models.
70
Reward Bench Score
60
GRM
50
SMORM-F
SMORM
Baseline-Single
Baseline-Multi
40
0.01
0.1
10
Weight Ratio
Figure 14: Hyperparameter analysis.
K INTERPRETATION OF WHY SORM FAIL IN OOD SETTING
0.4
0.6
Gollter Stare
0.2
0.5
Holptuness
Comcamness (=0 86)
0.0
Colarence 2-0 223
0.4
Complesty (e4.25)
0.2
Scores
Vorbosty
Proxy Score (= 0.55)
Helplates (-0 at)
0.2
Comerness (-0 98)
0.1
Complesty (= 0.39)
Verbootyy (--073)
0.0
10000
20000
20000
40000
10000
20000
30000
40000
Training Samples
Training Samples
(a)
(b)
Figure 15: Fine-grained attribute scores of the optimized policy model using (a) the baseline classifier and (b) SMORM-F as the proxy reward model.
To interpret the vulnerability of the baseline classifier to reward hacking, we adopt the same ex- perimental setting as in Sec. 3 and record evaluation scores for the generated responses across five dimensions: He felpfull pfulness, Correctness, Coherence, Complexite ty, and Verbos ty. These scores are derived using the multi-objeccivv head w of the trained SMORM. Figures 15 (a) and (b) illustrate the evaluation results when using the baseline classifier and SMORM-F as proxy reward models, respectively. In Fig. (a), employing the baseline classifier leads to improvements only in the Complexity ty and Ve Verbosity dimensions, while performance declines in the remain- ing attributes. Consequently, the generated responses are not considered high-quality by the gold reward model. In contrast, Fig. 15 (b) shows that using SMORM-F as the proxy reward model results in consistent improvements across all five fine-grained dimensions. These enhancements are also reflected in increased gold score. These findings indicate that conventional objeciveeeective reward model is typically insufficient to capture the multifaceted criteria that make a chosen response preferable rejected one.
38

---

Published as a conference paper at ICLR 2026
L ANALYSIS ON RM-BENCH
RM-Bench (Liu et al. 2025b evaluates reward models on two key dimensions: sensitivity to subtle changes and robustness to style bias. It includes three task types: (1) Easy: Chosen responses detailed and informative; rejected ones are concise and minimal. (2) Normal: Both responses share the same style but differ in key information. (3) Hard: Chosen responses are concise; rejected ones are detailed.
L.1 MORMS FALL SHORT ON EASY AND NORMAL TASKS.
In this section, we present empirical results to illustrate why baseline multi-objective reward models tend to underperform R.M- R.M-Bench (Liu et al. 2025b). We begin by evaluating selection of existing open- open-source single-objectiv and mblti-objective reward models that exhibit comparable performance on RewardBench (Lambert al. 2024). The evaluated models include:
Multi-objective reward models:
W
RLHF1oo/AAmoRM -laanaMM--aamaaa v0.
Single-ohjective reward models:
Ray2338/ccw--11mma--------s--::-
10
3. 1
Table 14: Comparison of models on Easy, Normal, and Hard tasks on RM-Bench.
Model Name
Easy
Normal
Hard
Avg
RewardBench
89.0
74.7
46.6
70.1
93.1
lkrcordoon/URK--Laa--3331--
84.0
73.2
53.0
70.0
92.9
84.6
72.2
50.2
69.0
89.4
Internalnn/neerrnnee-22-----aaa
82.6
71.6
50.7
68.3
90.2
Ray=335700a--LLmm3388-ffft11
83.5
72.7
48.6
kay:335/66kh-11mm33-----11111111
82.2
71.5
48.4
89.9
74.0
44.0
69.3
90.9
RLMB:11ww/A2mmm0--::1a3333------ WYDDAA/Neenotrro--3000--BB
82.5
70.8
50.1
67.8
90.4
81.0
71.4
56.1
69.5
92.0
The results of these comparisons are presented Table 14 For better illustration, we also visualize the results in Fig. [6] We observe that existing multi-objective reward models tend to underperform on the Easy and Normal tasks, even though their overall performance on RewardBench and the Hard tasks of RM-Bench is not among the worst. We attribute this phenomenon to the following: multi-objective reward models (MORMs) typically assess response quality through utility and style attributes. For example, HelpSteer2 (Wang al. 2024f) provides scores for correctness,
hvidia/Neemotron-44440--------
interlmm/ntemmn22220-cccua
NCSOrT7Laama33-00lsellll0
Ray233/CCMMlaaa4444--dnnnne
XrGmmdom/URMMHJJ.Maa333338
39

---

Published as a conference paper at ICLR 2026
Single-objective
74.5
Multi-objective
74.0
Normal Score
73.5
N
73.0
Ray2333//GRMllamm34BBBtttrr
72.5
72.0
inemiinnintemmin2222---ewewrrrr
71.5
Ray 2ty22333/GRM.llaam388-isssllll NIIDWWeemttron3330---eeea
71.0
82
84
86
88
90
Easy Score
Visualization of results RM-Bench. The size of each marker indicates the model's performance on the Hard task.
Figure 16:
helpfulnees, and coherence (utility), and complexity and verbosity (style). Formally, we define: U = [correctness, helpfulness, coherence], {verbosity, complexity J. The score margin is:
Ar
Wh Alu
Wverb Alverb + Weomp Alcomp
raistyle bias
Fu utiiiiy heads
In Easy tasks, longer chosen responses incur negative style penalties, potentially flipping Ar despite higher utility scores. In Normal tasks, responses should have similar style, implying Alverb Ahcomp s 0, but in practice, non-trivial deviations occur due to imperfect optimization. As shown in Appendix L.2 style noise degrades scoring accuracy and overall performance.
L.2 CASE STUDY AND WHY SMORM HELP
In this section, we present case study to interpret why baseline multi-objective reward models fall short on the Normal and Easy tasks of RM-Bench
We train multi-obe objective reward model (MORM) on the HelpSteer2 dataset and, for each attribute across all tasks, compute the mean and variance of the prediction differences-wiiee where each difference is defined as the chosen-response score minus the rejected-response score. The results are summarized in Table [5] As shown in the table, while the baseline MORM exhibits mean scores for complexity and verbosity that approximately zero across the Normal tasks, the corresponding variances are non-trivial. This indicates that, although paired responses should receive similar scores for complexity and verbosity, there remains considerable prediction bias these attributes in individual comparisons.
Table 15: Normalized pairwise differences per dimension (Mean (Variance))
Category
helpfulness
correctness
coherence
complexity
verbosity
Normal overall
0.08 (0.83)
0.00 (1.55)
0.07 (0.74)
0.17 (1.29)
0.00 (0.55)
Normal Correct
0.41 (0.75)
0.33 (1.48)
0.29 (0.88)
0.43 (1.31)
0.10 (0.52)
Normal False
-0.31 (0.65)
-0.38 (1.35)
-0.19 (0.46)
-0.13 (1.10)
-0.12 (0.55)
Hard overall
0.26 (1.02)
0.32 (1.90)
-0.28 (0.95)
-0.56 (1.85)
0.05 (0.81)
Hard Correct
0.78 (0.84)
0.85 (1.86)
0.01 (0.82)
-0.27 (2.24)
0.14 (0.81)
Hard False
-0.08 (0.84)
-0.03 (1.62)
-0.48 (0.93)
-0.75 (1.50)
-0.19 (0.77)
Easy overall
-0.11
-0.32 (1.93)
0.41 (1.38)
0.90 (2.36)
0.05 (0.75)
Easy Correct
(1.10(..1)
-0.07 (1.69)
0.61 (1.51)
1.12 (2.15)
0.12 (0.70)
Easy False
-0.76 (1.20)
-0.87 (2.02)
-0.03 (0.82)
0.40 (2.46)
-0.10 (0.83)
40

---

Published as a conference paper at ICLR 2026
Ideally, single-objective reward model (SORM) can better align with helpfulness and correctness, as its preference training dataset encompasses diverse attributes and tends to exhibit less sensitivity to superficial factors like response length. According to Theorem [ the aggregate score from the multi-objective reward model is lower-bounded by the single-objective reward score: I'm "2, y) = R Di=1 fo(r, v) 2 (x, v) E, where a; reflects the correlation between each attribute and the chosen/rejected preference direction. Consider scenario where response ya is preferred over yB according ground-truth labels. Initially, it may happen that I' (VA) (BB). especially when the style bias overwhelms the utility signal, i.c.. (vu) Tu (vB) < [ra(yA) - ro(y))). However, when the single-objective reward assigns higher score to y4, ie.. (,())) > r,(U)). the utility heads in the multi-objective model are progressively adjusted to favor yA- This adjustment increases the margin (r,(yA) ru (UB),, I[ eventually outweighing the bias introduced by stylistic components (ro). As a result, the style penalty's 's influence diminishes, and the model better reflects true preference judgments. To verify this, we train the SMORM using 40K samples from as Ds, and report the differences in Table [6] As shown, for normal cases, both the variance and verbosity are substantially reduced compared to the variance observed in helpfulness. This indicates that style-related biases no longer dominate the final decision, thereby enhancing the performance of the mujti-oh reward model on this task.
-eeied-Feebbbbe ack
Table 16: Normalized pairwise differences per dimension (Mean (Variance))
Category
helpfulness
correctness
coherence
complexity
verbosity
Normal overall
0.43 (0.62)
0.43 (0.68)
0.29 (0.41)
-0.02 (0.11)
0.01 (0.08)
Normal correct
0.73 (0.57)
0.71 (0.70)
0.53 (0.35)
-0.01 (0.13)
0.05 (0.10)
Normal false
-0.24 (0.09)
-0.17 (0.10)
-0.23 (0.11)
-0.03 (0.07)
-0.07 (0.04)
Hard overall
0.28 (0.70)
0.44 (0.93)
0.25 (0.55)
-0.30 (0.16)
0.33 (0.20)
Hard correct
0.77 (0.62)
0.91 (1.05)
0.68 (0.45)
-0.31 (0.17)
0.48 (0.22)
Hard false
-0.34 (0.11)
-0.16 (0.12)
-0.30 (0.15)
-0.29 (0.14)
0.14 (0.11)
Easy overall
0.57 (0.67)
0.43 (0.61)
0.34 (0.38)
0,26 (0.16)
-0.30 (0.20)
Easy correct
0.83 (0.56)
0.62 (0.59)
0.53 (0.30)
0.30 (0.17)
-0.31 (0.23)
Easy false
-0.26 (0.12)
-0.18 (0.16)
-0,27 (0.16)
0.14 (0.11)
-0.27 (0.12)
M THE USE OF LARGE LANGUAGE MODELS
Large Language Models (LLMs) were used to support the writing and editing of this manuscript. Specifically, an LLM assisted in improving clarity, refining phrasing, correcting grammar, and enhancing the overall readability of the text.
The LLM was not involved in the ideation, research design, data analysis, or development of any scientific content. All research questions, methodologies, and analyses were independently developed and executed by the authors.
41


Reviews:
[{"review_id": "yq8lAoTNPU", "reviewer": "Reviewer_9F27", "summary": "The paper introduces SMORM, a single-backbone with one Bradley–Terry (BT) reward head and one multi-objective regression head. The authors also provide theory proofs linking the two to explain the robustness gains. The architecture is novel while synthesizing insights from prior work (e.g., Armo, GRM). Empirically, SMORM improves scoring and out-of-distribution robustness and also alleviates reward hacking in RLHF.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well written and clearly positioned relative to prior work, while thoughtfully incorporating insights from it.\n\n\n* The method is simple and well-motivated, and the theoretical analysis makes the core idea more compelling.\n\n\n* The evaluation is thorough, covering both reward modeling and RLHF.", "weaknesses": "* The design of SMORL-L uses the mean of the multi-objective scores. However, using the mean does not make sense, which means you give all rewards equal weights given all prompts. Actually, this should be utilized in a contextual way and dynamically adapt to the user prompts [1][2].\n\n* The authors claim SMORM is flexible because the two heads are trained on different prompt–response pairs. In practice, this complicates using and coordinating two distinct datasets. I therefore disagree with the flexibility claim and suggest revising it.\n\n* Because the method uses two datasets, add a control that trains a BT reward model on the union of the multi-objective preference data (transferred to BT data using rules) and BT data. This would make the comparison fairer by isolating architectural gains from “more data” effects.\n\n* Based on question 2, I'm interested if not including new data if the conclusion still holds. This can be validated using a multi-objective preference dataset and transfer it to BT data using certain rules. Then you do the same ablations study comparing to baselines (multi-objective only, BT-only, combining them using two models,  and SMORM) trained on the same dataset. \n\n* Table 3 mixes base models: the paper evaluates with Llama-3.1-8B-Instruct but compares to ArmoRM on Llama-3-8B. Against contemporaneous baselines built on Llama-3.1-8B-Instruct (e.g., QRM-Llama-3.1-8B-v2, Skywork-Reward-Llama-3.1-8B-v0.2), the method shows no clear gains; notably, Skywork-Reward-Llama-3.1-8B-v0.2, trained only on Skywork80K (without multi-objective data), scores 93.1 on RewardBench. This weakens the claim.\n\n* Since the paper leverages multi-objective preference learning, Appendix A should include a more complete related-work section to cover this line of research [1–4].\n\n\n\n\n\n[1] Improving context-aware preference modeling for language models, 2024.\n\n[2] MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning, 2025.\n\n[3] Rethinking Diverse Human Preference Learning through Principal Component Analysis, 2025.\n\n[4] Personalizing reinforcement learning from human feedback with variational preference learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5KNqbcKHtC", "reviewer": "Reviewer_v1Q3", "summary": "This paper introduces SMORM (Single and Multi-Objective Reward Model), which jointly trains a Bradley–Terry (BT) single-objective head and a multi-objective regression head on a shared embedding space. The authors claim that this co-training improves robustness against reward hacking under out-of-distribution (OOD) conditions and enhances fine-grained multi-attribute reward estimation. The paper provides theoretical analyses connecting BT loss and regression loss through Fisher Information and MSE bounds and reports empirical results showing modest improvements over GRM, ODIN, and baseline classifiers on RewardBench, RM-Bench, and PPO/BoN experiments.", "questions": "1. Could the authors report standard deviations or confidence intervals across multiple seeds to verify statistical significance?\n2. Can the OOD analysis be explicitly decomposed into settings to substantiate the robustness claims (e.g., prompt-disjoint, response-disjoint, and mutual-disjoint)?\n3. Could the authors report per attribute ranking accuracy or correlation for helpfulness, correctness, coherence, complexity, and verbosity? It would be better, if authors include ablations that remove the multi objective head to isolate its contribution.\n4. Have the authors compared SMORM with simpler stabilization methods such as Batch Sum-to-Zero Regularization (BSR) under identical OOD conditions?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper offers a theoretically well-grounded contribution. The link between BT loss and regression loss via Fisher Information analysis is original and potentially influential.\n2. The joint single/multi-objective framework is conceptually neat and aligns with the broader goal of multi-dimensional reward alignment.\n3. The idea of embedding-space complementarity between preference learning and multi-attribute regression is interesting and could inspire follow-up research.\n4. Writing and structure are clear, and the motivation is intuitive.", "weaknesses": "1. Lack of statistical rigor.\nThe paper reports only mean scores without standard deviations or multiple-seed averages. Given the small performance margins (≈1–2 points on RewardBench or RM-Bench), these gains could easily fall within noise. This omission is especially problematic because the theoretical claim centers on variance reduction, yet no variance statistics are provided.\n\n2. Weak OOD validation and limited generalization coverage.\nThe paper claims robustness “in OOD settings” but evaluates only one prompt–response distribution shift. Unlike Hong et al. (ICML, 2025) (“On the Robustness of Reward Models”), which explicitly distinguishes in-domain (ID), prompt-disjoint, response-disjoint, and mutual-disjoint scenarios, this work tests only a single OOD split. No experiments examine robustness to unseen model families, novel prompt distributions, or response-generation biases, which would be expected to stress the model’s generalization. Consequently, the empirical evidence does not convincingly demonstrate the claimed OOD robustness.\n\n3. Fine-grained performance not well supported.\nAlthough the framework is motivated by multi-attribute supervision (helpfulness, coherence, verbosity, etc.), evaluation results remain at the aggregate level. No attribute-wise performance or correlation analysis is presented to verify the “fine-grained” improvement.\n\n4. Small effect sizes and lack of reproducibility details.\nReported improvements over GRM or ODIN are minor and not statistically supported. Also, RLHF results (PPO/BoN) appear based on single runs with no mention of seeds or error bars.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pVpqL3nULx", "reviewer": "Reviewer_xVES", "summary": "## Summary\nThe paper introduces **SMORM**, a joint reward-modeling framework that shares one backbone and trains (i) a Bradley–Terry (BT) single-objective head from pairwise preferences and (ii) a multi-objective regression head from fine-grained attribute scores. The method offers single-pass inference via three variants: **SMORM-F** (use the BT head only), **SMORM-L** (mean of attribute heads), and **SMORM-M** (average of BT and attributes).\n\n\nEmpirically, SMORM improves out-of-distribution (OOD) robustness for PPO and Best-of-N and boosts scoring on RewardBench and RM-Bench. The authors also report that, using the same multi-objective dataset, **a 7B SMORM model surpasses a 70B baseline** on RewardBench.", "questions": "1. **If Assumption (2) in Theorem 1 (positive-definite covariances) is relaxed to positive-semi-definite, do the results still hold?**  \n2. For the **7B > 70B** claim, a consolidated table with **confidence intervals** and multiple base models would better characterize robustness of this result across settings.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- **Clear OOD focus with concrete evidence.** The paper explicitly studies PPO/BoN under prompt-distribution shifts and shows baselines (e.g., GRM, ODIN) misspecify or overfit signals that lead to reward hacking, whereas SMORM variants maintain rising gold scores.  \n- **Principled link between BT and regression.** Lemma 1 upper-bounds expected BT loss by the regression MSE; Theorem 2 uses Fisher information to argue strictly better asymptotic MSE for both heads under joint training.  \n- **Practical single-pass design with strong results.** One shared forward pass and simple inference variants; results include RewardBench/RM-Bench gains and the 7B > 70B headline.", "weaknesses": "- **Assumptions (1–3) may be stringent in practice.** Theorem 1 requires (i) **bounded features**, (ii) **positive-definite** covariance matrices for BT and multi-objective features, and (iii) a **positive-correlation** condition via a coupling vector. In realistic learned embeddings, covariances can be **rank-deficient** (e.g., collinearity, limited per-attribute data, heavy regularization), and verifying the correlation condition empirically is non-trivial. Discussion or diagnostics on how often these hold in real RM pipelines would strengthen the claims.  \n- **Limited optimization-stability analysis for the joint objective.** The paper notes that “their joint training is non-trivial due to their fundamentally different forms,” but beyond asymptotic analysis and aggregate curves, there’s little on **training stability/convergence** (e.g., loss weighting sensitivity, gradient interference). An appendix gives some hyperparameter analysis, yet a focused study on training stability (or convergence) is still missing.\n- Missing a related work  \"On the Robustness of Reward Models for Language Model Alignment\", which focuses also on the OOD setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aWhcucIKIK", "reviewer": "Reviewer_u8z6", "summary": "This paper investigates the problem of reward hacking in reinforcement learning from human feedback, particularly under out-of-distribution (OOD) conditions. The authors propose SMORM, a unified framework that jointly trains a Bradley–Terry (BT) single-objective reward model and a multi-objective regression-based reward model using a shared embedding space and provided theoretical support on the connection between the BT loss and the regression objective and highlight their complementary benefits.", "questions": "1. Under the BT model, rewards are identifiable only up to an additive constant. In Theorem 1, the two sides may implicitly live on different scales. How do you resolve the shift when coupling BT and regression heads?\n\n2. The shared embedding f typically lives in high dimensions; in practice its covariance matrices can be singular. Many proofs assume positive definiteness. If this fails (e.g., rank deficiency due to collinearity), what precisely degrades?\n\n3. With massive over-parameterization, the Fisher Information Matrix  for LLM parameters is typically singular. Page 24’s asymptotic variance claims appear to rely on an invertible FIM is unlikely to hold. Thus, it requires further clarification.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-structured and clearly presented\n\n2. The idea of co-training BT and multi-objective regression heads under a shared embedding space is new and well-motivated.\n\n3. The proposed methods is supported by empirical evaluations, under both in-distribution (ID) and out-of-distribution (OOD) settings.", "weaknesses": "1. While the joint-training idea is new, the method itself (two linear heads sharing embeddings) is architecturally simple and appears as a simple combination of existing method. Therefore, in my view, the main novelty lies in the theoretical analysis of complementary benefits of BT and multi-objective reward modeling.\n\n2. The theoretical results rely on several strong and unverifiable assumptions, such as positive-definite covariances and global positive correlation between single-objective and multi-objective reward signals. In practice, it is unclear whether these assumptions are satisfied in noisy environments.\n\n3. No code provided currently.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)