Reward Model Routing in Alignment
Authors: Xinle Wu
Yao Lu
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-armed bandits
Preference optimization
Reward model
Online DPO
Primary area: foundation or frontier models, including LLMs
TL;DR: A hybrid reward model routing framework that improves alignment in LLMs.

Abstract: Reinforcement learning from human or AI feedback (RLHF/RLAIF) has become the standard paradigm for aligning large language models (LLMs). However, most pipelines rely on a single reward model (RM), limiting alignment quality and risking overfitting. Recent work explores RM routing—dynamically selecting an RM from a candidate pool to exploit complementary strengths while maintaining \(O(1)\) RM calls—but existing methods suffer from cold-start and insufficient exploration. We propose {\name}, a hybrid routing framework that combines offline RM strengths learning with online Bayesian selection. In the offline stage, a multi-task router is trained on preference data to estimate per-RM reliability. In the online stage, a Bayesian Thompson sampling router performs per-query RM selection, initializing RM-specific weight vectors with offline embeddings as Gaussian priors and adaptively updating their posteriors with online rewards to adapt to the evolving policy distribution. Extensive experiments on instruction-following (AlpacaEval-2, Arena-Hard, MT-Bench) and reasoning (GSM8K, MMLU) benchmarks show that {\name} consistently outperforms individual RMs, RM ensembling, and existing routing methods.

REWARD MODEL ROUTING IN ALIGNMENT
Xinle Wu, Yao Lu
National University of Singapore {wuxl, Luyao)@0omm nus edu. sg
ABSTRACT
Reinforcement learning from human or AI feedback (RLHF/RLAIF) has become the standard paradigm for aligning large language models (LLMs). However, most pipelines rely on a single reward model (RM), limiting alignment quality and risking overfitting. Recent work explores R.M ddynmmiclll selecting an RM from candidate pool to exploit complementary strengths while maintaining ((1) RM calls-but but existing methods suffer from cold-start and insufficient ex- ploration. We propose BayesianRouter, a hybrid routing framework that combines offline RM strengths learning with online Bayesian selection. In the offiin stage, multi-task router is trained on preference data to estimate per-R.M reliability. In the online stage, a Bayesian Thompson sam- pling router performs per-query RM selection, initializing RM-specific weight vectors with offline embeddings as Gaussian priors and adaptively updating their posteriors with online rewards to adapt the evolving policy distribution. Extensive experiments instruction-folloving (AlpacaEval-22, Arena-Hard, MT-Bench) and reasoning (GSM8K, MMLU) benchmarks show that BayesianRouter consistently outperforms individual RMs, RM ensembling, and existing routing methods.
1
INTRODUCTION
Large language models (LLMs) have revolutionized artificial intelligence, demonstrating substantial capabilities in language understanding, reasoning, and open-ended text generation across diverse domains (Guo et al. 2025 Achiam et al.) 2023). To safely and effectively deploy L.LMs, recent post-training techniques, particularly reinforcement learning from human feedback (RLHF) and its Al-augmented variant (RLAIF) (Wang et al. 2024b). aimed at aligning LLMs with human values and preferences. These methods fine-tune LLMs to internalize nuanced human preferences, bridging the gap between raw pretrained performance and user-aligned behavior. In standard RLHF, reward model (RM) provides the feedback signal for optimizing the policy LLM; for example, in PPO, the RM provides a scalar reward to directly increase the probability of preferred responses (Ziegler et al.) 2019). while in direct preference optimization (DPO), the RM compares two candidate responses to determine which is better (Dong et al. 2024).
Recent RLHF/RLAIF pipelines often rely on a single RM throughout training (Kaufmann et al., 2024). This design choice, however, can be suboptimal due to (1) limited generalizability: no single RM consistently excels across all tasks, evidenced by benchmarks like RewardBench 2 (Malik et al. 2025). An RM tuned for one type of content (e.g. conversational helpfulness) may perform poorly on a different genre (e.g. mathematical reasoning), Ieading to suboptimal alignment when one fixed RM is used universally; (2) high costs: using a powerful general-purpose LLM (e.g. GPT-5) as the RM can provide high-quality feedback, but the cost of querying such a model at scale is prohibitive (Zheng et al. 2023). In practice, this makes large RMs impractical for extensive RLHF training, and (3) risk of overoptimization: relying on a single RM amplifies the risk of overfitting to that RM's idiosyncratic biases or noise, which can lead the policy to exploit the RM's flaws (i.e., reward hacking) rather than truly align with human intent (Coste et al. 22023; Zhang et al. 2024). Collectively, these issues undermine the robustness and scalability of single-RMo alignment.
To overcome these challenges, recent studies have started to leverage an ensemble of reward models, combining the strengths of multiple RMs. Prior work in (Coste et al., 2023) Zhang et al., 2024) explored multi-RM, approaches, but naively using multiple RMs in parallel for every query is extremely costly and can introduce conflicting or noisy signals when the models disagree. Among ensemble strategies, routing methods are particularly interesting: instead of aggregating all models' outputs, router dynamically selects the most suitable RM for each input, preserving the benefits of model diversity while minimizing overhead. In this spirit, LASER (Nguyen et al.) (024) is, to our knowledge, the first method to apply instance-level RM routing in RLHF. LASER frames reward model selection a contextual multi-armed bandit problem. During baO-based RLHF training, for each batch of prompts, a bandit
'Code is available http://iithbbboom//iiineWaaaaags

---

(LinUCB) chooses a single RM from a candidate pool to label the policy's 's responses; the policy is then updated on this preference-labeled data, and the router is updated based on the policy's resulting reward signal. By selecting one RM at time, LASER avoids the overhead of running all RMs and adapts the choice of RM as training progresses.
While LASER showed the promise of adaptive RM selection, important limitations remain: (1) coarse- grasse-grinedddeee routing: LASER selects one RM per batch of prompts, whereas prompts within the same batch may favor different RMs. This batch-level routing often makes suboptimal choices for many individual queries; (2) limited exploration: using LinUCB (which relies on point estimates and optimism), the router can prematurely lock onto suboptimal RM and insufficiently explore others. In other words, LASER's 's bandit may over-exploit one arm without adequately probing alternatives that could be better for certain query types, and (3) inefficient cold-start: at the start of training, LASER assumes all RMs are equally good and must gather many interactions to identify each RM's unique strengths. This slow start results in suboptimal RMs being used in early training, which reduces sample efficiency and makes the outcome sensitive to initial conditions.
To address these issues, we propose Royes,an a hybrid RM routing framework that integrates learned model of RM strengths, paired with an online Bayesian selection strategy to accelerate routing at a small compute overhead. Specifically, Bayesian consists of (1) an offline router with language model- based encoder that is trained on existing preference datasets to predict which RMs will perform better for given query. We use a multi-tash task objective: Bradley-Terry ranking head scores each candidate RM, and a classification head predicts whether each RM would choose the better answer in given pair. This offline router captures each RM's specialization in a shared embedding space, providing a rich prior for selection; (2) online Bayesian router: during RLHF fine-tuning, BayesianRouter employs Bayesian Thompson sampling for instance-level RM selection. The router treats the query embedding as context and maintains a Gaussian posterior for each RM's reward model. For each query, it samples a reward estimate for each RM from its posterior and selects the RM with the highest sample, then uses that RM's feedback to train the policy and update the posterior. By sampling from an uncertainty aware model, instead of relying on a single deterministic estimate, the router naturally balances exploration and exploitation and can more robustly discover which RM is optimal for each query type. We initialize the online router using the prior knowledge from the offline router to inherit the knowledge on each RM's strengths, as well as to bootstrap the cold-start. As training proceeds, the router updates this knowledge and adapts to the evolving policy distribution while retaining the offline insights.
We evaluate Bayesian on both destruction-folloiing benchmarks (including AlpacaEEval2 and MT-Bench) and academic benchmarks (including GSM8K and MMLU). The results demonstrate that BayesianRouter significantly outperforms strong baselines, such as the single best RM, RM ensemble methods, and LASER.
2 RELATED WORK
LM-based Reward Model. Language model-based reward models (RMs) act as proxies for human preferences and play a central role in RLHF and RLAIF (Kaufmann et al., 2024 Wang et al. 2024b). They are commonly categorized into three families: classifier RMs (Liu et al.) 2025), generative RMs (Yu et al. (0225)), and LLM-as-a-judge (LAJ) (Hurst et al., 2024). In early RLHF pipelines, RMs typically provided a scalar reward for each (prompt, response). which was then optimized with policy-gradice
methods such as PPO (Kaufmann et al. 2024). More recent RLAIF approaches often use RMs to conduct pairwise comparisons between responses and apply objectives such as DPO to encourage the policy to prefer the better response (Guo et al. 2024). Research on RMs primarily focuses on improving the reliability of reward signals. For example, [Wang et al. (2024a) filter unreliable preference data by comparing rankings across training iterations; Yu et al.) (2024) adopt divide-and-conquee strategy that decomposes response evaluation into simpler claim-level judgments; and Liu et al. (2025) build a large-scale dataset of 40M preference pairs via human-Al collaboration, enabling smaller RMS to outperform much larger models. Orthogonally, R.M ensembling schemes) has been shown to improve robustness and mitigate overoptimization risks (Coste al. 2023 Zhang et al. 2024). In addition, benchmarks such as RewardBench, RM-Bench, and RewardBench provide systematic evaluations of different RMs across domains, offering practical guidance for model selection (Lambert et al. 2024 Malik et al. 2025 Liu al. 2024b).
(c.g., averaging, nower-confidence bound, or weteertainttwwwigheeee
Routing LLM Queries. Research on routing LLM queries has so far mainly focused on LLM inference, where the aim is to assign each query to the most suitable model before decoding in order to balance accuracy and efficiency, For instance, Lu et al. 2023) train router using reward-model based scores of candidate responses as supervision; Ding et al. (2024) adaptively switch between cloud and edge models depending on query difficulty; Ong et al.] (2024) propose ROUTELLM, which employs classifiers (e.g., matrix factorization, causal LLM classifier) to decide whether a query should be routed to a strong or weak model: [Shadid et al.) (2025) analyzes L.M performance on benchmark tasks, clusters user queries by similarity, and dynamically routes each query to the berf-performing LLM for its cluster,
2

---

achieving higher accuracy at lower cost compared to trained routers; and Frick et al. (2025) introduce P2L, which uses Bradley-Terry modeling and sparse pairwise preference data to train routers that scale to hundreds of candidate L.L.s. While these methods are designed for inference scenarios, to the best of our knowledge, there has not been work on leveraging offline preference data to pretrain a router specifically for reward models, which differ from inference routers in terms of imput features and label construction. Moreover, a purely offline router is prone to out-of-distribution (OOD) issues, and may generalize poorly when deployed on unseen data distributions.
Multi-Armed Bandits (MABs). MABs offer a classical framework for sequential decision-making under uncer- tainty, balancing exploration and exploitation by pulling one arm per round and observing stochastic feedback (Zhoul 2015; Bouneffouf & Rish, 2019). This formulation naturally fits routing, where each query is assigned to one model with only partial feedback available. Among contextual bandit algorithms, Lin (Li et al.) 2010) and Bayesian linear Thompson sampling (Agrawal & Goyal 2013) are two widely used approaches: the former uses optimism via confidence bounds, while the latter samples from a posterior enable uncertainty-awar exploration. Beyond these. variants like KL-UCB (Garivier & Cappé, 2011). OFUL (Abbasi- Yadkori et al.) 2011). or logistic/GLMM bandits (Filippi et al., 2010) can also be applied depending on feedback type and distributional assumptions.
MAB algorithms have recently been applied to LLM inference routing. For example, 2025) formulates model selection as a contextual bandit problem, training sefference-conditonn dynamic routing policies on offline data and leveraging model identity embeddings to generalize across architectures, thereby enabling adaptive selection of high- performance, low-cost LLMs at inference time. In the reward modeling setting, the only existing MAB-based router is LASER (gguyen al. 2024), which leverages LinUCB to select one reward model per input batch during RLAIF training. In contrast, our BayesianRouter replaces point- further integrates offline-learned priors to address both exploration inefficiency and cold-start limitations.
point-estimate estimate exploration with Bayesian posterior sampling, and
3 METHODS
In this section, we introduce BayesianRouter, reward model (RM) routing framework designed for adaptive RM selection within preference- based alignment pipelines. BayesianRouter is designed for the DPO family, while it could in principle also benefit reinforcement learningbbased methods such as PPO that rely on scalar reward signals, although evaluating its performance in that setting is beyond the scope of this work. Concretely, for each input prefer- ence cois-consisiing of prompt and two candidate responses from the policy model- BayesianRouter selects the most appropriate RM from a candidate pool to evaluate the pair. The resulting preference signal is then used to train the policy model via online DPO. An overview of Boyesian is shown in Figure D
We structure this section as follows. Section 3.1 briefly reviews the standard online DPO algorithm, formally de- fines the RM selection problem, and provides overview of BayesianRouter. Section 3.2 introduces the offline RM router, which leverages preference datasets and a multi-task objective to model RM strengths, i.e. identifying which candidate RM is most reliable for given preference pair. Section 3.3 presents the Bayesian Thompson sam- pling-based online router, which adaptively selects RMs during online DPO training and updates its posterior distri- bution with policy feedback. Finally, Section [344 describes how we integrate the offline-learned RM strengths with the online router to alleviate the cold-start problem.
3.1 PROBLEM FORMULATION AND METHOD OVERVIEW
Online DPO Training Pipeline. We follow the standard online Direct Preference Optimization (DPO) (Guo et al. 2024, Dong et al., 2024) setup to iteratively align the policy model æ. Let ne denote the model at training step t, initialized from a pretrained policy To. At each step, the model receives mini-batch of prompts { and for each
24 sample i candidate responses y {vl... ..... , } ~ Tm (-12z)) A reward model R. evaluates these candidates to construct preference pairs: for classifier RMs, R(I;, nf) outputs a scalar score and we form a pair (2a, V." v/)
if R(x; y > R(a;, v!); for generative RMs, the model directly compares two responses (ui vu ) and indicates which is preferred, yielding preference pair of the same form. Collecting over the batch gives a preference dataset D Pprof- [(x;, v!"" b/))e--
The policy n, is then updated on Drre using the DPO loss RRaaallov et al. 2023). which encourages the policy to increase the likelihood ratio between the preferred and dispreferred responses relative to a frozen reference model mee
LDPO= =
Σ
log o Boog
(
x))
B log
(
xx)
(1)
ID [proot
Teet (y x)) Tree xx)
(x.00 JE pref
3

---

Offline router
Online DPO
(x,y+,y-)
update parameters
CLS
BT
Policy
Responsel
RM
Response2
Prompt
P
yl
1+
DPO
loss
Responsel
x
Response2
y2)
y2
MLP
Online router
RM pool
sample
encoder
prior
00
BayesianRouter
(x.y1.y2)
update online router
Figure I: Overview of Route..
where Tref is typically set to To. (()) is the logistic function and / is scalar hyperparameter. Without loss of generality, we the standard DPO objective, though other variants such IPO (Azar et al. 2024 and SLiC (Zhao et al. 2023) are also compatible with our framework.
Problem Formulation. In the online DPO pipeline, the choice of the reward model (RM) directly shapes the mini- batch preference dataset per constructed at each step, thereby determining the quality of the alignment signal. Empir- ically, no single RM uniformly dominates across preference types or domains. For example, RewardBench (Malik) et al., 2025) reports that Skywork Reward- -2-LLama Llama-222 3.1 8B ranks first overall but, while outperforming the second-place LMJnit- -weunntttwen..... 727 on Math and Safcty preferences, it underperforms LMJnit qwwn2 5-:e2..5777 72b Factuality. To exploit complementary strengths across N RMs, one option is ensemble them (e.g., majority vot- ing over multiple RMs). However, this increases the per-step inference cost from O(1) to O(N) RM calls, resulting in a significant increase in training cost. In contrast, adopt per-query routing: for each unlabeled preference pair (xi, Vi Ve ), we select a single most suitable RM to annotate it. This preserves ((1) RM calls per step while still leveraging complementary strengths of N RMs.
Formally, let M = {R,, J be the candidate RM pool. We aim to learn a router Router(:; W) (parameterized by
W) that maps unlabeled preference pair (æ,, vl v. to RM:
Rn Router v/. vi W).
The selected RM yields more reliable preference annotations for the current batch, thereby providing higher-quality supervision for policy updates.
Routers can be offline (trained static preference corpus and kept fixed during online DPO) online (updated during training using feedback from the current policy). Offline routers can leverage existing labeled offline preference datasets but may fail under distribution shift between the offline training data and the online data; online routers can adapt to the target distribution but suffer from cold-start and exploration challenges early in training.
Our Approach:
BayesianRouter (Overview). We propose BayesianRouter, a hybrid RM routing framework that couples an offline, RM strengths learning stage with an online, distribution adaptation stage (see Fig. [ The offline-learned RM strengths provide a high-quality initialization that mitigates the cold-start issue and improves early batch-level routing accuracy; the Bayesian online updates adapt the router to the evolving data distribution and policy, The following subsections detail the offline router, the online RM router, and the integration strategy.
3.2 OFFLINE RM ROUTER
Given a candidate of reward models M (R,, and an offline preference dataset D Ppref = {(z, yos V/. H 1 where l; E {0, }} indicates which of ys and y/ is the preferred response to the prompt the goal of the offline router is to predict, for new unlabeled preference pair (at, y, v²), which RM most likely correctly identify the preferred response. Figure (left) illustrates the architecture of our offline RM router.
4

---

Collecting RM Behavior Data. To construct the training signals for the offline router, we first collect the behavior of each candidate RM on D pref. Concretely, we run each RM R. on each preference pair = (Ti, Vs, v.) D and record a binary indicator 6()) {0, 1} that equals 1 if Rn agrees with the ground truth & and otherwise, yielding m. n N).
beb =[( (1
Preference-pair feature construction. Unlike LASER in (guuyn et al. 2024) that uses only the prompt as router input, we encode the whole preference pair (I4, Yt, v/)) because an RM's decision depends not only the prompt but also on the semantic content of the two responses and their contrast. Concretely, we first concatenate the prompt with each response and encode them with shared pretrained encoder Enc(:; W.):
e; Enc(#;
ys; We),
e = Enc(a; || 1/: W.).
We then aggregate these encodings into a single preference -air representation by
h MT.P (e; e/; le; e<]]: Wi) R",
where | denotes vector concatenation, | is the element-wise absolute difference, and (1 is single-layer MLP used to fuse features.
Based on the preference feature h, we adopt a multi-task objective with two prediction heads. The primary head is Bradley-Terry (BT) head, which assigns ability score each RM such that, given preference pair, the RM with the highest score selected as the most reliable one. The auxiliary head is a classification (CLS) head, which independently predicts for each RM whether it can correctly identify the preferred response in the given pair.
BT head (primary). We define a disagreement sample as a preference pair on which two RMs produce different behavior labels. Such samples capture the relative competence of the two RMs and are therefore suitable for training Bradley-Terry (BT) head to predict per- RM ability scores. Formally, from Deeh we extract the disagreement set
Dbe {(6, n, n') 8(m) 1, g(n²) 0}- We learn an embedding matrix Fbt Nxd whose n-th row represents RM R,, We compute BT scores as inner products, i..., s (h; [n][ and sy = (h;, E [n'),, and optimize the pairwise logistic (Bradley-Terry) loss
Cut
Σ
log ogo - -s
[Dotl
(g.,,,"')ED.
This objective encourages the BT head to assign higher ability scores to RMs that win paired comparisons.
CLS head (auxiliary). Since the BT head relies only on disagreement samples, it ignores the information contained in the remaining portion of Dech- To better exploit the full dataset, introduce auxiliary per-RM binary classification head. Given a preference pair gi, the CLS head predicts each candidate RM's behavior label 6!n) from the preference embedding h,. Concretely, we learn an embedding matrix Erts E N whose n-th row corresponds to RM Ra, compute logits z = (h,, Fecs((n) for every RM, and optimize the binary cross-entropy over Deeh:
Cola
Σ
(55h)) log (2!") + (1 - "(")) log (1 (l-o((+))) cz
(3)
[Dohhl
By independently predicting each RM's behavior, the CLS head provides complementary supervision to the pairwise BT objective, benefiting the BT ranking through shared representation learning.
Training objective and offline output. The router is trained by minimizing the combined loss Lestal Lst + Cels: where À controls the contribution of the auxiliary classification loss. We optimize the encoder parameters We, the MLP parameters Wi, and the head parameters Ebt and Eels After training, we retain the BT embedding matrix Ebt as the prior for online routing, since it captures relative RM strengths conditioned on preference pairs.
3.3 BAYESIAN ONLINE RM ROUTER
Unlike the offline router, which is trained on static preference data and remains fixed during online training, the online router is continuously updated after each routing decision using observed rewards. By adapting to the evolving policy- induced distribution preference pairs D pref. the online router mitigates distributional mismatch that would otherwise limit the effectiveness of the offline router.
In online routing, only the supervision from the selected RM is observed for each preference pair, making the problem natural instance of contextual partial-feedback learning (i.c., contextual bandit). Here, candidate RMs correspond
5

---

to arms, the preference pair serves as the context, and the problem can be addressed with contextual multi nume bandit (MAB) algorithms. (Nguyen et al. 2024) used LinUCB for online routing. However, we empirically find LinUCB often collapses to a single fixed arm after a few batches, likely because per-arm observations are scarce and contexts are similar, which leads to premature exploitation. To encourage continued exploration and to more reliably discover which contexts each RM specializes in, we adopt Bayesian Thompson sampling.
Bayesian Thompson Sampling We model the expected utility of selecting RM R. on a preference-pair embedding h, with Bayesian linear regression:
wn h; E, ~~~(0,
where w, E IR is a latent weight vector for R., and o² denotes observation noise. Each RM maintains Gaussian posterior Wn ~ ~N(mp: Σn) that is updated only when R, is selected. At training step t, given a batch of preference- pair embeddings KEB, we perform Thompson sampling by drawing a sample from each RM's posterior for cach preference pair:
w(()) ~ NV( )
ny = arg max h w(()
and the router selects Ray for that pair. Let I((² = li B. n. n denote the indices in the batch assigned to RM Ra; after observing scalar rewards (n)ier! for these pairs, we update R, posterior using the accumulated sufficient statistics of its assigned pairs:
x((11)
(>(())
-1 +
1
Σ hch!
)
(5)
a²
iEZ!!
1 ((+1)
((++1) (())
-1 1
(0)
Σ
f h1)
(6)
σ²
When no offline prior is injected we initialize μ and Σ (00) = a Ja, where of is the prior variance. Let Liro be the DPO loss on preference pair labeled by R, We follow LASER to take the raw reward to be Pt pPO We use quantile normalization to normalize the raw reward and obtain the final variance-reducee and numerically stable reward Pr, (details in Appendix B
While pr, = - might implicitly favor "casy" RMs that align with the current (potentially flawed) policy, this choice justified by two factors:(i) Robustness to Overfitting: "flattering" " but low-quality RM (e.g.. favoring short sequences) can only maintain low loss within a narrow distribution. As Thompson Sampling (TS) explores diverse RMs, the resulting policy updates quickly shift the distribution away from such biased regions, causing the flawed RM's loss to (ii) Consistency as Proxy for Quality: High-quality RMs typically exhibit self-consistent preference gradients, leading to training trajectories with lower variance and lower overall loss magnitude compared to inconsistent, low-quality RMs [Shen et al.] (2023; Liu et al. (2024a). By treating rewards as time-series input to the Bayesian posterior, the router learns to favor RMs that provide long-term stable guidance rather than short- term agreement. Our empirical results further validate that this signal effectively identifies specialized RMs without collapsing to sub-optimal "yes-man" models.
3.4 WFFLINE-ONLNG INTEGRATION
While the offline and online routers can each perform RM routing as defined in Section 3.1 both have intrinsic limitations. The offline router leverages abundant supervised preference data but may degrade under distribution shift, whereas the online router adapts to the policy-induced distribution but suffers from cold start and exploration challenges. Thus, neither component alone is sufficient in practice.
To address this, our key idea to combine their complementary advantages. naïve approach is to directly combine their outputs (e.g., by weighted averaging of their predicted scores), but such schemes require a manually tuned global weight whose optimal value is unclear and may vary across training stages. Instead, we propose a more principled strategy based on prior injection. The insight is that both the offline BT head and the online Bayesian router can be viewed as linear models over the preference-pat embedding: the offline BT head computes (h, Eb (r]) where Evg [n] is the learned RM embedding, while the online router computes (h, wx) where w,, is the latent RM weight vector. The semantic roles of Epc [n] and w, are thus closely aligned, differing mainly in the source of supervision (offline labels versus online rewards). This motivates initializing the online Bayesian router with the offline BT embeddings.
Concretely, we set the prior mean of each RM's weight vector in Eq. 4 to the corresponding offline embedding, i.e., Nn.
(0) = En]] This initialization provides the online router with prior knowledge about which types of preference
6

---

pairs each RM is likely to handle well. As result, it mitigates the cold-start problem and improves early routing accuracy. During training, the posterior distributions are iteratively refined using online rewards, allowing the router to adapt to the evolving policy-induced distribution while retaining the offline prior regularizer. In this way, BayesianRouter combines the robustness of offline training with the adaptivity of online learning.
4 EXPERIMENTS AND RESULTS
We evaluate Boyesia on instruction -oollwiinn-oolowiigg and reasoning benchmarks with the goal of demonstrating that it enables more effective reward model selection and consequently leads to superior alignment performance.
4.1 EXPERIMENTAL SETUP
Models. We initialize the policy model with LLaMA3-SFT-v2 released by Dong et al. 2024 The reward
model (RM) pool consists of N = small yet high-performing models from the RewardBench leaderboard (Malik) Sattrall-R--fo-RAF-----
(RM_0), GRM-Llama)) rdmo rowade et al. 2025): -ww.--eeaaadoddee---tttttt. (RM.2), and Skywork- Rew-rd-waraa V2 Qwen3-0. 6B (RM33.. For the GRM gww.aemmmmaaeemmm
(RM_1),
offline router encoder, we use Smo SmolLM2 135M Instruct (see Appendix A.I)
Datasets and Metrics.
Offiine preference datasets. To train the offline router, we combine two human-annotated preference datasets:
HelpSteer3 (Wang et al. 2025) and RM-Bench (Liu et al. 2024b), resulting in 50,402 preference pairs.
Instruction-following benchmarks. Following (Dong et al. 2024). we evaluate the instruction-follont following ability AlpacaEval-2 (Dubois et al.] 2023), MT-Bench (Zheng al. 2023), and Chat-Arena-Hard (Li et al. 2024)
Policy models are trained on the iterat iveeiiiepppe prre-ppt-------- ieer3 20K prompt set released by (Dong et al. 2024), and evaluated with length controlled AlpacaEval (Dubois al. 2024), where model responses are compared against the SFT baseline using GPT-4 as the judge.
Reasoning benchmarks. Following (Nguyen et al., 2024). we evaluate performance under different training distri- butions by training and testing on two reasoning benchmarks: GSM8K (Cobbe et al. (021) and MMLU (Hendrycks) We report accuracy for both datasets, with detailed statistics provided Appendix A.
et al.
2020).
Baselines.
We compare Bayesian against the following methods:
Single RM: Use a fixed RM from the pool for preference annotations.
Majority vote: Annotate each preference pair with RMs and select the final label via majority voting.
Random router: Randomly select an RM to annotate a preference pair.
Optimization (UWO): This ensemble method (Coste et al. 2023) down-weights prefer- ence pairs that exhibit high disagreement among the RMs. We implement this by setting the weight for each pair to its consensus rate (i.e., the fraction of RMs that agree with the majority preference).
Uncertainty-Weigee
LASER: The first RM routing method that employs LinUCB to select single RM per batch (Nguyen et al. 2024)
w/o offline: Variant of BayesianRouter without offline priors.
w/o online: Variant of BayesianRouter that uses only the offline router for RM selection.
4.2 MAIN RESULTS
Table
summarizes the performance of Bayesian against baseline methods. We have the following key ob- servations: (1) BayesianRouter consistently outperforms all basclines both instruction-followin and reasoning benchmarks. Given that these results are achieved after training on datasets with distinct distributions, it demonstrates the adaptability of BayesianRouter diverse types of training data. (2) BayesianRouter surpasses the performance of single RM baselines, showing that dynamically routing among multiple candidate RMs effectively aggregates their complementary strengths. In practice, users often choose single RM based leaderboard performance, which may not accurately reflect true RM performance across tasks or domains. Our results show that BayesianRouter eliminates this reliance and even surpasses the best-performing RM identified in hindsight. (3) BayesianRouter significantly outperforms the Majority Voting and UWO ensemble methods. While ensemble methods can also lever- age complementary RMs, they require O(N) RM calls per query, making them impractical for scaling to large RM pools. In contrast, BayesianRouter achieves higher performance with only 0(1) RM calls. BoyesianRouter also substantially outperforms the routing baselines Random Routing and LASER, further validating the effectiveness of our routing strategy. (4) BayesianRouter outperforms its two ablations, w/o offline and w/o online, highlighting
7

---

Each prompt in Re- the complementary contributions of the offline-leameed prior and the online Bayesian feedback loop. Removing ei- ther component leads to notable degradation. Notably, the w/o offline variant exceeds LASER, showing that our per-query Bayesian Thompson sampling router is superior to LASER's per-batch LinUCB approach.
Table 1: Main results instruction and reasoning benchmarks.
Instruction-Folloving
Reasoning
Method
AlpacaEval-2
MT-Bench Chat-Arena-He Hard
GSM8K
MMLU
SFT
50.00
50.00
50.00
67.63
54.29
RMO
56.02
52.50
59.60
72.78
56.00
RMI
61.86
56.25
64.80
74.22
57.03
R.M2
59.50
53.75
63.20
73.92
56.57
RM3
60.37
52.50
62.00
74.53
56.28
Majority vote
60.75
53.75
63.40
74.22
56.71
Random router
58.39
52.50
61.20
73.46
56.07
UWO Coste et al. 2023)
61.74
56.25
63.60
74.30
56.43
LASER (guuyen et al. 2024)
60.50
51.25
62.40
74.00
56.35
w/o offline
60.99
53.75
63.20
74.37
56.64
w/o online
61.61
57.50
64.40
74.68
56.85
Boytei
63.23
58.75
66.20
75.66
57.39
4.3 ADDITIONAL ANALYSIS OF BayesianRouter
Effectiveness of Offiine Router We evaluate the offline router's ability route preference pairs the most suitable RM and analyze the factors affecting its performance. For in-distribution (ID) evaluation, we use the official test split of the HelpSteer3 dataset; for -fitt--diittiuutionn (OOD) evaluation, we adopt RewardBench wardBench 2 is paired with one preferred response and three rejected responses, which we flatten into pairs, discarding all ties. We further filter both test sets to retain only those samples where least one candidate RM correctly identifies the preferred response, yielding 1,723 ID and 2,939 OOD preference pairs. Under this setup, oracle router achieves 100% accuracy. better understand the router's behavior, we introduce two additional base- lines: w/o CLS, which removes the classification head from the offline router, and 0.5B encoder, which replaces the SmoILM2-135M-Inntee Instruutttt encoder with 10e.mm2.---.B-IIsttttt substantially outperforms single-RM, majority voting, and random routing baselines in the ID setting, and also deliv- ers consistent gains in the OOD setting, though with smaller margins. Nevertheless, there remains a considerable gap from the oracle, highlighting both the effectiveness and the generalization challenges of offline routing-likely due limitations the scale, diversity, or domain coverage of available preference data. This underscores the importance of Routeisn online adaptation. In addition, removing the classification head leads to performance degradation, validating the benefit of multi-objectivy training. Finally, while larger encoders yield modest improvements in routing accuracy, adopt the 135M encoder as practical balance between performance and efficiency.
Table reports the results. Overall, our offline router
Table 2: In-distribution and Out-of-distribution performance comparison.
Method
In-distribution
Out-of fnt-rf-distributiin
Score
Factuality
Precise IF
Math
Safety
Focus
All
RM_O
77.54
75.44
59.22
78.76
85.10
75.61
77.61
RM_1
81.43
84.33
68.44
81.18
96.00
95.73
87.65
RM
79.51
80.04
67.38
79.03
97.60
90.85
85.88
RM.3
81.14
77.64
70.92
90.05
92.70
92.38
85.34
Majority
83.17
77.74
67.73
85.75
96.50
90.85
85.64
Random
79.80
79.94
65.96
82.80
92.80
87.80
84.48
Ours (w/o CLS)
89.73
84.12
66.67
85,22
96.20
92.99
87.34
Ours (135M)
90.31
84.85
65.60
86.83
96.20
92.07
87.92
Ours (0.5B)
90.77
85.16
66.31
87,90
95.90
91.46
88.06
8

---

Controlled simulation of online DPO. In practical online DPO training. it infeasible to obtain real-time human annotations for policy-generated responses, making it impossible to directly verify whether the RM selected by a roter produces correct preference labels. To address this, we design controlled simulation using the 2,939 human- labeled preference pairs from RewardBench pairs and let the router select an RM label them. Since ground-trule labels are available, the chosen RM provides the correct annotation, thereby directly assessing the quality of routing. Table [ compares BayesianRouter with its ablations. The results show that BayesianRouter achieves the highest annotation accuracy during training and attains the best downstream alignment performance. This confirms that BayesianRouter's gains originate from accurate RM routing rather than other confounding factors. The overall decrease in performance compared to the main results is attributed to the limited number of training samples. See more results in Appendix
Instead of sampling responses from a live policy, we replay existing
measure how often
Table 3: Controlled online DPO results.
Method
AlpacaEval-2
MT-BBench Chat-Arena- Hard
GSM8K
MMLU
Acc.
w/o offline w/o online
55.78
54.84
55.87
67.78
54.61
85.68
56.72
56.41
57.71
68.39
54.82
87.92
BayesianRouter
57.63
56.76
58.15
68.76
54.93
88.23
Ablation on integration strategy. To validate the effectiveness of combining the offline and online routers, we compare BayesianRouter with simple variant, Weighted-score, For each preference pair, Weighted-score computes two separate score vectors; from the offline router's Bradley-Terry head, representing each RM's estimated compe- tence, and $2 from the online router initialized with zero-meat priors, representing the RM's current reward estimates. To address scale differences, each score vector is converted into probability distribution via softmax. The two dis- tributions are then combined using a fixed weight a as 8 = 0s + (1 - a))22, and the RM with the highest combined score is selected. We sweep (0.25, 0.5, 0.75) and report the best-performing setting. Table presents the results. RoyesianRouter consistently outperforms the weighted-score variant across all datasets. This demonstrates that ini- tializing the online router with offline BT embeddings provides principled and more effective mechanism to integrate offline knowledge with online adaptation, rather than relying on simple linear weighting scheme.
Table 4: Ablation studies on integration strategy.
Method
AlpacaEval-2
MT-Bench
Chat-Arena-HH Hard GSM8K
MMLU
Weighted-score
61.12 63.23
56.25 58.75
63.80
74.37
56.75
Ours (BayesianRouter)
66.20
75.66
57.39
Training Efficiency. We evaluate the training effi- ciency Boyesian Unlike majority-voting en- sembles that require O(N) RM calls per preference pair, BayesianRouter selects single RM (1) cost. The extra overhead relative to single-RM baseline comes from (i) encoding preference pairs with the offline router and (ii) lightweight Bayesian posterior updates. Both are independent of the number and size of candidate RMs and amortize as the pool grows. Moreover, if the router often selects smaller, cheaper RMs, the overall cost may even fall below using single large RM. To demonstrate the scalability of Bayusia we increase both the size and number of candidate RMs and compare wall- clock training time on the GSM8K dataset against four baselines: the fastest single RM, the slowest single RM, Majority Voting, and LASER. Figure 2 that while BayesianRouter is slower than the fastest single RM, it substantially outperforms Majority Voting and the slowest single RM, demonstrating its efficiency. LASER runs marginally faster than BayesianRouter because reuses policy embeddings rather than encoding preference pairs independently. Detailed experimental settings provided in Appendix A.I
Fastest RM
Training Time (hours)
25
Slowest RM
20
Majority Voting
LASER Bayesian
15
10-
5
0
GSMBK
Figure 2: Training efficiency.
9

---

5 CONCLUSION
In this work, we addressed the problem of adaptive reward model (RM) selection in iterative DPO training pipelines. We proposed Baytsia hybrid framework that first learns multi-task offline router capture RM strengths from preference data, and then injects this prior knowledge into Bayesian Thompson sampling-based online router. The resulting framework adaptively selects single RM for each preference pair while continually refining its routing policy through online rewards. Extensive experiments show that BayesianRouter consistently suraases single-RM methods, RM ensembles, and strong routing baselines, demonstrating its effectiveness. For future work, we plan to design routers that jointly optimize the trade-off between annotation accuracy and RM inference cost to further improve RLHF alignment under constrained computational budgets.
REFERENCES
Yasin Abbasi-Yadial Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In International conference on machine learning, pp. 127-135. PMLR, 2013.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In Interna- tional Conference on Artificial Intelligence and Statisties, pp. 4447 4455- PMLR, 2024.
Djallel Bouneffouf and Irina Rish. A survey on practical applications of multi- aulti-armed and contextual bandits. arXiv preprint 1rXiv:1904.1000 2019.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek. Jacob Hilton, Reicchiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overopti- mization. arXiv preprint arXiv:2310.02743, 2023.
Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks VS Lakshmanan, and Ahmed Hassan Awadallah. Hybrid Ilm: Cost-efficient and quality-aware query routing. arXiv preprint arXiv:2404. 14618, 2024.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.0760 2024.
Yann Dubois, Chen Xuechen Li. Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba. Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36:30039-30069, 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint wrXiv:2404.04477 2024.
Sarah Filippi, Olivier Cappe, Aurélien Garivier, and Csaba Szepesvári. Parametric bandits: The generalized linear case. Advances in neural information processing systems, 23, 2010.
Evan Frick, Connor Chen, Joseph Tennyson, Tianle Li, Wei-Lin Chiang, Anastasios N Angelopoulos, and Ion Stoica. Promp--oolleaderboard. arXiv preprint arXiv:2502. 14855, 2025.
Aurélien Garivier and Olivier Cappé. The kl-ueb algorithm for bounded stochastic bandits and beyond. In Proceedings of the 24th annual conference on learning theory, pp. 359-376. JMLR Workshop and Conference Proceedings, 2011.
10

---

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mea- suring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.2/276, 2024.
Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback. 2024.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403. 13787, 2024.
Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandle approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pp. 661-670, 2010.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline. Blog post. (Accessee 07-02-2025), 2024.
Yang Li. Llm bandit: Cost-efficient Ilm generation via prefereece--conditioned dynamic routing, arXiv preprint arXiv:2502.02743, 2025.
Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang. Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, et al. rewadr:- Scaling preference data curation via human-ai synergy. arXiv preprini arXiv:2507.0135 015,2,,,,,, 2025.
Yan Liu, Xiaoyuan Yi, Xiaokang Chen, Jing Yao, Jingwei Yi, Daoguang Zan, Zheng Liu, Xing Xie, and Tsung- Yi Ho. Elephant in the room: Unveiling the impact of reward model quality in alignment. arXiv preprint arXiv:2409.1902 2024a.
Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. Rm-bench: Benchmarking reward models of language models with subtlety and style. arXiv preprint arXiv:2410.161/ 16184, 2024b.
Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the
expert: Efficient reward-guided ensemble of large language models. arXiv preprint arXiv:231 1. 1.08692, 2023.
Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A Smith, Hannaneh Hajishirzi, and Nathan
Lambert. Rewardbench 2: Advancing reward model evaluation. arXiv preprini arXiv:2506.01937. 2025.
Duy Nguyen, Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. Laser: Learning to adaptively select reward models with multi-armed bandits. arXiv preprint arXiv:2410.01735, 2024.
Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. Routellm: Learning to route I1ms with preference data. arXiv preprint arXiv:2406. 18665, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In seventh Conference on Neural Information Processing Systems, 2023. URL htthstts:: /wtp/aaxiivv/..
http://wwwwv..caaa////////oo org/ /200..23055.0 18290
Ahmad Shadid, Rahul Kumar, and Mohit Mayank. Ori: O routing intelligence. arXiv preprint 10005 2025.
Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng, Haitao Mi, Daniel Khashabi, and Dong Yu. The trickle-down impact of reward (in-) consistency on rlhf. arXiv preprint arXiv:2309. 16155, 2023.
Zhaoyang Wang, Weilei He, Zhiyuan Liang, Xuchao Zhang, Chetan Bansal, Ying Wei, Weitong Zhang, and Huaxiu Yao. Cream: Consistency regularized self-rewarding language models. arXiv preprint arXiv:2410. /2735, 2024a.
11

---

Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang- Bo Mao, Sitaram Asur, et al. comprehensive survey of Ilm alignment techniques: Rihf, rlaif, ppo, dpo and more. arXiv preprint arXiv:2407. 16216, 2024b.
Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Hoo- ShooChhng Shin, Felipe Soares, Alexander Bukharin, Ellie Evans, Open human-annotatee preference data across diverse
Yi Dong, and Oleksii Kuchaiev. Helpsteer3-preferee tasks and languages. arXiv preprint arXiv:2505. 1475, 2025.
Jiachen Yu, Shaoning Sun, Xiaohui Hu, Jiaxu Yan, Kaidong Yu, and Xuelong Li. Improve llm-as-a-judge ability as a general ability. arXiv preprint arXiv:2502.11689 2025a.
Qingchen Yu, Zifan Zheng. Shichao Song, Zhiyu li, Feiyu Xiong, Bo Tang, and Ding Chen. xfinder: Large language models automated evaluators for reliable evaluation. In The Thirteenth International Conference Learning Representations, 2025b. URL
ew. net. fo rum? ra TUU0ku
Tianyu Yu, Haoye Zhang. Yuan Yao. Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning milms through open-source ai feedback for super gpt-4v trustworthiness. arXiv e-prints, pp. arXiv-2405, 2024.
Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen, Zhiqing Sun, and Chuang Gan. Improving reinforcement learning from human feedback with efficient reward model ensemble. arXiv preprint arXiv:2401.16635, 2024.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likeli- hood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. Judging aa--as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 042:46595-466623 2023.
Li Zhou. A survey on contextual multi-armed bandits. arXiv preprint
2015.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08550 2019.
A EXPERIMENTS
A.I EXPERIMENTAL SETTINGS
Training setup.
as the encoder of the offline router and perform full-parameter fine-tuning using AdamW with learning rate of 2 x 10-5, batch size 8, and A 0.2. We train for 2 epochs with weight decay 0.01.
Offline router training. We adopt
Online policy training. The policy model is initialized from LLaMA3-SFT-v2. We fine-tune for epoch using LoRA (rank 16, = 32) applied to the q-pro and v.pro projection matrices. We use learning rate of 5 x 10 with Adam. Each prompt batch contains 16 prompts; for each prompt we sample candidate responses at temperature 0.8 and construct 4 preference pairs, yielding 64 preference pairs per batch. We set the parameter 33 in Equation to 1. We do not use prompt templates during training or inference. The maximum input length is 128 tokens and the maximum output length is 256 tokens, except for the second turn of MT-Bench, where the input length is increased to 512 to accommodate multi-turn context. When initializing the online router with offline- learned embeddings, we set the prior variance for the RM weights of 0.02 (with noise variance r² 1). reflecting stronger trust in the prior; when using the online router alone, we set of = with prior mean and g² 1.
All experiments are conducted on a server equipped with 8x RTX A6000 GPUs, each with 48GB memory.
Benchmark Details. For AlpacaEval-2,, MT-Bench, and Chat-Arena-Hard, we use them exclusively as test sets. The details are as follows:
AlpacaEval-2 (Dubois et al. 2023): This a single-turn dialogue benchmark consisting of 805 prompts covering a wide range of topics.
12

---

MTT-Bench (Zheng et al. 2023): This is multi-turn dialogue benchmark comprising 80 prompts across various domains. Each prompt contains two questions: the model first answers the initial question, then receives the initial question, its response, and the second question input to produce second response. The evaluation is conducted based on the quality of both responses jointly.
Chat-Arena-Haal (Li al. 2024): This benchmark contains 500 high-quality prompts selected from user queries in Chatbot Arema. It is designed to evaluate models on creativity, data analysis, deep comprehension, and problem- solving abilities.
For GSM8K and MMLU, we split each dataset into training and test sets. Policy models are fine-tuned on the training split and evaluated on the held-out test set. For GSM8K, accuracy against the ground-truth labels. For MMLU, we use xF i nnder (Yu et al.) 2025b) to extract the predicted choices before comparing them with the labels. Table 5 summarizes the number of instances in each split.
math verify parse model responses and compute
Table 5: Dataset statistics.
Dataset
Train
Test
GSM8K 7465
1319
MMLU
11233
2809
Settings for Efficiency Analysis. To evaluate the scalability of BayesianRouter, we extend the RM pool to 8 candidates. In addition to the RMs used in the main experiments, we select 4 larger models from the RewardBench 2 leaderboard (Malik et al., 2025): RIsE-Judge-Owen22 323 (RM.4), http://woo-rriittlaateeeaee (RM 5), Selene-1-Min
1-BB
1B (RM 6), and RISE-Juddge-Wwnnd 5-7B (RM
We measure training time on 8 GPUs. For BayesianRouter, Majority Voting, and LASER, we allocate resources as follows: RM 4 occupies 2 GPUs; RM 0, RM 5, RM 6, and RM 7 each occupy GPU; RM 1, RM and RM 3 share 1 GPU; and policy training uses the remaining GPU. For the single-RM baselines, we consider the slowest RM (RM44) and the fastest RM (RM.3). In the slowest-RM setting, RM.4 is assigned GPUs with data parallelism for policy training. In the fastest-RM setting, RM.3 occupies GPU with data-parallel policy training.
B METHOD DETAILS
MAB reward normalization. To provide stable learning signals the bandit router, do not directly use per-pair losses as rewards. Instead, for each training step we first compute a batch-level baseline over all preference pairs in the batch:
4,
Σ (t),
B
EEB
where B, denotes the set of preference pairs at step t and (t) is the training loss pair under its selected RM. The instantaneous advantage-style reward for pair is then
r((() E, -
which normalizes for batch difficulty and highlights the relative quality each pair within the batch.
Because the scale of r4) may drift over time, following LASER (Nguyen al. 2024). we further apply quantile- based rescaling. Let R1:0-1 = (ry(F) < f, E } denote the set of past rewards up to step t - We compute the empirical th and 80th percentiles of this set, denoted of and ah The normalized reward then
if (t) < to if r > hi otherwise.
A(t)) =
rt( to q
This two-stage procedure--hacch--asseee batch-baseline centering followed by quantile variance-reduced and numerically stable across the training process.
rewards that are both
13

---

Table 6: Controlled online DPO results.
InstructionFoollovee
Reasoning
Method
AlpacaEval-2
MT-Bench
Chat-Arena-Han
GSM8K
MMLU
Acc.
SFT
50.00
50.00
50.00
67.63
54.29
RM0
53.05
53.33
53.39
67.17
54.18
77.61
RMI
56.14
55.88
57.93
68.16
54.79
87.65
RM2
55.83
54.55
56.32
67.70
54.54
85.88
RM3
55.28
54.05
55.60
67.55
54.33
85.34
Majority
55,66
54.29
56.92
67.40
54.47
85.64
Random
53.55
53.13
54.51
67.25
54.18
84.48
UWO (Coste et al. 2023
56.19
55.56
56.65
67.78
54.61
85.64
LASER (Nguyen et al. 2024
55.20
55.17
54.96
67.40
54.40
85.54
w/o off.
55.78
54.84
55.87
67.78
54.61
85.68
w/o on.
56.72
56.41
57.71
68.39
54.82
87.92
Royee
57.63
56.76
58.15
68.76
54.93
88.23
Table Ablation studies on reward design.
Method
AlpacaFval-2
MT-Bench
Chat-Arena-Hard
GSM8K
MMLU
Acc.
Full advantage
56.50
56.52
57.51
68.46
54.79
87.72
Light_advantage
56.12
55.81
56.13
67.85
54.72
87.38
Ours (w/o offine)
55.78
54.84
55.87
67.78
54.61
85.68
C ADDITIONAL EMPIRICAL RESULTS
Controlled online DPO results. Table 6 presents presents the complete controlled online DPO results, using the same train- ing setup as described in Appendix A.I
Analysis of reward design. We further analyze the effect of different reward formulations for online routing. In particular, we design two alternative variants that compute rewards hased on RM-to-RM comparisons rather than batch-level normalization.
Full Advantaee For each preference pair, all candidate RMs are queried to obtain their induced training losses. The router still selects single RM via Thompson sampling. but the reward is defined a binary advantage: if the selected RM's loss is no greater than the average loss across all RMs, the reward is set to 1, otherwise to 0. This design removes the confounding effect of sample difficulty and purely reflects the relative quality of RMs.
Light-ade Light_advantage. As a more scalable compromise, we randomly sample C' = RMs to compute the baseline average, instead of evaluating the full pool. This reduces computational overhead while still approximating the comparative signal.
We compare these two variants against our proposed nortalzorm reward (with quantile rescaling) under the controlled online DPO setup on RewardBench 2. Table 7 reports the results. As expected, the Full advantage variant achieves the strongest performance, since it leverages the most informative RM comparisons, but the cost of prohibitive computation and poor scalability. The Light_advane sagganaadv variant attains performance close to the Fulltadvant while being more efficient, showing that subsampling can preserve much of the benefit. Our proposed batch-normalize reward is less competitive in isolation, but it is by far the most efficient and, when combined with the offline prior, yields the best overall alignment performance.
Empirical Justification for the Linear Head potential concern regarding our architecture whether linear model is sufficiently expressive to capture the complexities of reward model (RM) selection, which might involve high-order feature interactions. We argue that since our router operates on embeddings produced by aligned LLM--based encoder, the non-linear semantic structures are already internalized within the representation space. To empirically validate this, we conducted a diagnostic study the offline setting. We froze the pre-trained encoder and compared the performance of two different prediction heads: (i) Linear head, utilizing a standard Bradley-Tery (BT)
14

---

regression layer; (ii) Non-linear head, consisting of a single-layer MLP (with ReLU activation) followed by the BT layer.
As shown in Table 8 the non-linear head yields only marginal improvements over the linear head in both In- Distribution (ID) and Oft-o--Distribution (OOD) scenarios. The fact that significantly more expressive non-linear head fails to provide substantial gains suggests that the encoder successfully organizes the preference-pait embeddings into a largely linearly separable space for the RM-selection task. Consequently, linear head is not an oversimplifica- tion but sufficient and computationally efficient choice for both offline prediction and online Bayesian updates.
Table 8: Comparison of Linear and Non-linear (MLP) prediction heads top of frozen LLM embeddings. Results are averaged over 3 independent runs.
Output Head
ID Accuracy (%)
OOD Accuracy (%)
MLP BT head
91.41 0.10
88.42 0.06
Linear BT head (Ours)
91.31 = 0.09
88.35 + 0.04
C.1 SUFFICIENCY OF UNIMODAL POSTERIOR UNDER DISTRIBUTION SHIFT
A critical assumption in our Bayesian Linear Thompson Sampling (BLTS) is that the uncertainty in RM selection can be adequately modeled by a unimodal Gaussian posterior. One might wonder if the shifting policy distribution during online training could induce complex, multi-modal uncertainty that a single Gaussian cannot capture. To investigate this, we compared BLTS against more expressive Online Bootstrap Thompson Sampling variant. This baseline maintains M = 10 independent linear heads; at each step. it randomly selects one head for routing and updates each head using Poisson(1) 1) resampling weights approximate a non-parametric posterior. This approach is generally more robust in capturing multi-modal or complex reward distributions.
As shown in Table 9 BLTS consistently outperforms the Bootstrap TS variant across both reasoning (GSM8K) and general knowledge (MMLU) benchmarks. The observation that more flexible posterior family offers additional gain-and even slightly underperforms- into representation space where the uncertainty remains well-behaved. We attribute this to the high-quality and diverse preference data used during the offline pre-training phase, which allows the encoder to collapse complex multi-modal semantic structures into stable emhedding space that remains effective even as the policy evolves.
that the task-aligned encoder successfully maps preference pairs
Table 9: Comparison between Online Bootstrap Thompson Sampling and our Bayesian Linear Thompson Sampling (BLTS).
Method
GSMSK
MMLU
Bootstrap TS (M 10)
74.15 74.37
56.43
BLTS (Ours)
56.64
D LLM USAGE
We acknowledge the limited use of large language models (LLMs)) as an editorial assistant to enhance the clarity, grammar, and overall linguistic quality our manuscript. Specifically, an LLM was employed for minor stylistic improvements and grammatical corrections, well as to refine sentence structures for better readability. No LLM was used for content generation, idcation, methodological design, experimental execution, or data analysis. All scientific content, intellectual contributions, and experimental results presented in this paper are solely the work of the human authors. The authors take full responsibility for the entirety of the paper's content, including any text that may have been subjected to LLM-assisted polishing.
15


Reviews:
[{"review_id": "69qMKqeALp", "reviewer": "Reviewer_nfhC", "summary": "The paper addresses a key limitation in aligning large language models (LLMs): the reliance on a single reward model (RM) in pipelines like Reinforcement Learning from Human Feedback (RLHF). Using a single RM can lead to overfitting, reward hacking, and poor generalization across diverse tasks.\n\n\nTo solve this, the authors propose the Bayesian Router. This novel hybrid framework dynamically selects the most suitable RM from a candidate pool for each query during Direct Preference Optimization (DPO) training.\n\n\nThe method consists of two main stages:\n\n\n* Offline Router Training: An offline model is first trained on existing preference datasets to learn the \"strengths\" or reliability of each RM in the pool. This router encodes the full preference pair (prompt, response 1, response 2). It uses a multi-task objective, combining a Bradley-Terry (BT) head (to learn relative RM abilities from disagreement samples) and a Classification (CLS) head (to predict an RM's correctness on any given sample).\n\n* Online Bayesian Selection: The learned offline embeddings (specifically from the BT head) are used as a Gaussian prior to initialize an online Bayesian router. This online router, based on Thompson sampling, performs instance-level (per-query) RM selection during the DPO process. It adapts to the evolving policy by updating its posterior distribution using rewards derived from the DPO loss .\n\n\n\n\n\nThe paper claims that this hybrid approach effectively solves the \"cold-start\" problem (via the offline prior) and poor exploration (via Thompson sampling), achieving state-of-the-art results. Experiments on instruction-following (AlpacaEval-2, MT-Bench) and reasoning (GSM8K, MMLU) benchmarks show that Bayesian Router outperforms single RMs, ensemble methods, and the previous SOTA routing method, LASER.", "questions": "1. As mentioned in the \"Weak Points,\" the online reward $\\hat{r}_n^i$ derived from $-\\mathcal{L}_{DPO}^i$ seems confounded. Could a \"good\" but \"difficult\" preference from a high-quality RM be unfairly penalized with a high loss (low reward) simply because it is far from the current policy? Could you elaborate on the justification for this signal and discuss why it doesn't just reward RMs that are \"easy\" for the policy?\n2. In Appendix C, you introduce a \"Full Advantage\" reward (based on comparing the selected RM to all other RMs) and show it outperforms the w/o offline variant. This seems like a more robust (though expensive) reward. How does this \"Full Advantage\" reward perform when combined with the full Bayesian Router (i.e., with the offline prior)? Is it possible that the main results are constrained by the heuristic reward, and not the router itself?\n3. Could you provide more details on the practical overhead of the offline training stage? Specifically, what was the computational cost (e.g., in GPU-hours) to (a) generate the $\\mathcal{D}_{beh}$ dataset by running all 4 RMs over the 50k offline pairs, and (b) train the offline router, relative to the cost of the main online DPO training run? This would help in assessing the method's practical utility.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Well-Motivated Problem: The paper tackles a well-known and significant problem in LLM alignment—the overoptimization and brittleness of single-RM pipelines. The goal of efficiently leveraging a diverse pool of RMs is a clear and valuable research direction.\n* Novel & Principled Method: The hybrid offline-online design is the paper's key strength. It correctly identifies the weaknesses of purely offline approaches (distribution shift) and purely online approaches (cold-start, inefficient exploration). Using the offline-learned BT embeddings as the prior mean $\\mu_n^{(0)}$ for the online Thompson sampler is an elegant and principled way to integrate both components.\n* Strong Design Choices: The method improves upon prior work (LASER) in several well-justified ways:\n    * It uses instance-level routing, which is more granular than LASER's batch-level routing.\n    * It employs Thompson sampling for better, uncertainty-aware exploration, directly addressing the potential for premature exploitation in LinUCB (used by LASER).\n    * It encodes the full preference pair $(x, y_1, y_2)$ rather than just the prompt $x$, which is a more informative context for RM selection.\n\n* Comprehensive Experiments: The evaluation is exceptionally rigorous.\n    * It uses a good mix of instruction-following and reasoning benchmarks.\n    * It compares against a strong set of baselines, including single-best RMs, ensemble methods (Majority Vote, UWO), and the main competitor (LASER).\n    * The ablation studies are excellent, clearly demonstrating the individual contributions of the offline prior (BR vs. w/o offline) and the online adaptation (BR vs. w/o online).\n\n* Insightful \"Controlled Simulation\": The analysis in Section 4.3 (Tables 3 and 6) is a standout. By using a dataset with ground-truth labels (RewardBench 2), the authors directly measure the router's annotation accuracy and show it correlates with downstream performance. This provides strong, direct evidence that the router is actually selecting the correct RM and that this is the source of the performance gains.", "weaknesses": "* Practical Complexity: The proposed system is significantly more complex to deploy than the baselines. It requires a full, separate pre-training stage to (1) run all $N$ RMs over a large offline dataset to create $\\mathcal{D}_{beh}$, and (2) train the multi-task offline router. This represents a non-trivial barrier in terms of computation and engineering.\n* Confounding Online Reward Signal: The reward signal for the online router, $\\hat{r}_n^i$, is derived from the negative DPO loss, $-\\mathcal{L}_{DPO}^i$. This signal seems potentially confounded. An RM that strongly challenges the current policy (e.g., by correctly identifying a \"preferred\" response that the policy currently disfavors) might induce a high DPO loss, resulting in a low reward for the bandit. Conversely, a weak or \"yes-man\" RM that agrees with the policy's (potentially flawed) preferences might get a low loss and a high reward. The paper does not fully justify why this heuristic signal is optimal, although the empirical results suggest it works.\n* Scalability of the RM Pool: The main experiments use a pool of $N=4$ RMs, and the efficiency analysis uses $N=8$. It is unclear how the framework scales to a much larger pool (e.g., $N=50$). While $O(1)$ inference cost is maintained, the offline training (especially the BT head, which relies on disagreement pairs) and the online update of $N$ posteriors could become bottlenecks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RyenqgHmD0", "reviewer": "Reviewer_Bb3B", "summary": "This paper addresses the limitations of using a single, fixed Reward Model (RM) in alignment pipelines (RLHF/RLAIF), such as limited generalization and the risk of overoptimization. While RM ensembles improve robustness, they incur high computational costs (1$O(N)$ calls).2 Existing routing methods (like LASER) attempt to maintain $O(1)$ cost but suffer from cold-start issues and insufficient exploration", "questions": "In addition to the weakness section, I have the below questions:\n\nHow do you justify using the negative DPO loss as the reward signal for the bandit? Given that this reward signal measures the policy's ability to learn a preference (right or wrong), rather than the correctness of the RM's preference, how does the proposed feedback loop ensure the selection of high-quality RMs and avoid reinforcing easily-learned but incorrect preferences?\n\nThe \"Full Advantage\" reward design mentioned in Appendix C seems more conceptually aligned with measuring RM quality than the negative DPO loss, although it is more expensive. Why was this conceptually stronger reward signal not adopted as the primary method, perhaps using the \"Light Advantage\" approximation to manage the cost?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper tackles an important problem. The proposed hybrid approach, integrating offline pre-training with online adaptation, is in the context of RM routing and directly addresses limitations of prior work (cold-start in LASER, distribution shift in purely offline methods).\n\nRecognizing the alignment between the offline Bradley-Terry head and the online Bayesian router (both acting as linear models over the preference-pair embedding) allows for the effective injection of offline embeddings as priors. This is empirically validated as superior to heuristic weighting schemes.", "weaknesses": "The most critical weakness lies in the definition of the reward signal used to update the online Bayesian router (Section 3.3). The reward is defined as the negative DPO loss: $\\tilde{r}_n^i = -\\mathcal{L}_{DPO}^i$ (L303). The DPO loss measures how well the policy aligns with the preference provided by the RM. A high reward (low DPO loss) indicates that the policy already agrees with the RM or can easily model that preference.Crucially, this is not a measure of the RM's quality or the correctness of the preference itself. If an RM provides an incorrect preference, and the policy easily learns this incorrect preference, the bandit receives a high reward. This methodology conflates the ease of policy alignment with the quality of the alignment signal, creating a perverse incentive that does not encourage the selection of the most reliable RM, and may instead accelerate confirmation bias or reward hacking.\n\nInconsistent Empirical Results and Exaggerated Claims: The paper claims that \"Bayesian Router consistently outperforms all baselines on both instruction-following and reasoning benchmarks\" (L354-356). This claim is demonstrably false according to the results presented in Table 1.On AlpacaEval-2, Bayesian Router (63.23) is outperformed by Majority vote (63.40) and UWO (63.60).On MT-Bench, Bayesian Router (58.75) is significantly outperformed by the single RM1 (64.80), UWO (61.74), and even Random router (61.20).Furthermore, based on the SFT baseline scores (54.29 for GSM8K, 67.63 for MMLU), it appears the columns for GSM8K and MMLU might be swapped in Table 1. If this is the case, the performance of Bayesian Router on MMLU (57.39) is drastically worse than RM1 (74.22), UWO (74.30), and LASER (74.00).The proposed method fails to consistently beat the strongest baselines, significantly undermining the central claims of the paper.\n\nThe experiments utilize a small pool (N=4) of relatively small, homogeneous RMs (ranging from 0.6B to 3B parameters, L328). The benefits of routing are typically most apparent when the pool includes diverse models with clear specializations (e.g., safety vs. reasoning) or vastly different scales. The limited diversity makes it difficult to assess the router's ability to effectively exploit complementary strengths.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SlP4XyRwcI", "reviewer": "Reviewer_xgBK", "summary": "The paper proposes BayesianRouter, a hybrid reward-model (RM) routing framework for preference-based alignment (DPO family). Experiments on instruction following (AlpacaEval-2, MT-Bench, Arena-Hard) and reasoning (GSM8K, MMLU) claim consistent gains over: best single RM, ensembling (majority, UWO), and the prior routing baseline LASER", "questions": "1. Judge LLM and length-controlled AlpacaEval are used for instruction following evaluation. Would this risks judge-model bias? Results should report variance across judges.\n2. Were their hyperparameters tuned per RM ($\\beta$, LoRA rank, sampling temperature), or held constant—potentially under-optimizing some RMs?\n3. For the computation cost, the router adds encodings + bandit updates; wall-clock plots are described, but exact GPU hours and per-step RM latency distributions (small vs large RMs) would clarify cost/quality trade-offs.\n4. Offline router shows a non-trivial gap in OOD case; the paper argues online adaptation bridges it, but I don’t see stress tests where online distribution shifts during training. Is it possible to include such an experiment study?\n5. Since reward is a function of policy loss, how stable is routing across training (do posteriors collapse to 1–2 RMs)? Please include posterior traces $(\\mu,\\Sigma)$ over steps and the share of traffic per RM if possible.\n6. With larger pools (>8 RMs), do we still get O(1) advantages in practice? Any diminishing returns or exploration debt? (You mention scaling, but numbers are brief.)\n\nOverall I think the paper tackles a real pain point in RLHF/RLAIF with a clean method and interesting results. Addressing my above concerns could further strengthen the paper.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Clear problem framing: RM routing is a compelling middle ground between costly O(N) ensembles and single-RM training.\n2. Using offline BT embeddings as priors for a Bayesian linear bandit seems a pretty interesting design.\n3. Main results across five benchmarks show better performance over existing works\n4. The paper argues O(1) RM calls with lightweight extra overhead, and compares wall-clock training to single-RM, LASER, and majority vote", "weaknesses": "1. To my understanding, the main novelty of this work is the prior injection from offline embeddings into online TS. The contribution is solid engineering + good fit to RM routing, but may feel unsurprising conceptually.\n2. The online update uses (normalized) negative DPO loss per routed pair as the reward. This couples router training tightly to the current policy’s loss landscape and may conflate pair difficulty and RM quality.\n3. Concerns on evaluation choices & fairness (see questions for details)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i4XpAvgG4x", "reviewer": "Reviewer_PxRA", "summary": "The authors propose a Bayesian approach to online reward model selection to improve LLM performance.\nThe approach combines offline and online data. With offline data, a neural network representation of preference pairs alongside two prediction heads for prompt-response preference pairs is trained. Based upon this model, the reward model with best prediction performance for a given prompt-response preference pair is selected.   With online data, the authors use a Bayesian linear regression model of reward model \"utility\" (based upon the representation obtained offline). A sample of reward models from the posterior distribution is used to select the model better suited for a given a prompt-response preference pair (Thompson-sampling).\nThe proposed approach is evaluated on experimental testbed with alternative reward model selection models such as majority opinion, uncertainty-weighted optimization and linear upper confidence bound (as in Nguyen et al., 2024).", "questions": "1. I encourage the authors to provide arguments to support ad-hoc modeling choices (linear model of utility, robustness of neural representation for different tasks, Gaussian noise).\n\n2. I find the alternative model selection models may not provide a competitive benchmark. Perhaps the linear upper confidence bound proposed in Nguyen et al., 2024 duly extended to account for preference pairs might be a more decent comparison.\n\n2. Thompson sampling optimizes immediate expected gain under a single posterior sample. This can lead to suboptimal long-term exploration (over-exploitation) and inefficient learning when preferences are high-dimensional or multimodal.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed approach aims to provide the RLHF training paradigm with the ability to adapt according to changing tasks while minimizing additional computational effort.\n\n2. The numerical results indicate the proposed that reward model selection mechanism outperforms alternative approaches for model selection.", "weaknesses": "1. The choice of a linear model for utility is presented in ad-hoc fashion. A linear model has significant limitations for example it fails to capture interactions between features (e.g., \"helpfulness is only valued if tone is polite\") or hierarchical semantics (e.g., \"if response is correct, then prefer shorter length; otherwise, correctness dominates\").\n\n2. The performance of the linear model of utility relies heavily on the representation quality (which is obtained offline).  Any attempt to jointly re-train representation and utility will take away the simplicity of Bayesian updating of a linear-gaussian model.\n\n3. The choice of Gaussian noise is also ad-hoc. One can not represent multimodal preference beliefs (e.g., two equally plausible user preference patterns) with such model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)