Ask a Strong LLM Judge when Your Reward Model is Uncertain
Authors: Zhenghao Xu
Qin Lu
Qingru Zhang
Liang Qiu
Ilgee Hong
Changlong Yu
Wenlin Yao
Yao Liu
Haoming Jiang
Lihong Li
Hyokun Yun
Tuo Zhao
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
RLHF
Reward Models
LLM-as-a-Judge
Primary area: deep_learning
TL;DR: We propose an uncertainty-based routing framework that efficiently complements a fast RM with a strong but costly LLM judge.

Abstract: Reward model (RM) plays a pivotal role in reinforcement learning with human feedback (RLHF) for aligning large language models (LLMs). However, classical RMs trained on human preferences are vulnerable to reward hacking and generalize poorly to out-of-distribution (OOD) inputs. 
By contrast, strong LLM judges equipped with reasoning capabilities demonstrate superior generalization, even without additional training, but incur significantly higher inference costs, limiting their applicability in online RLHF. 
In this work, we propose an uncertainty-based routing framework that efficiently complements a fast RM with a strong but costly LLM judge. Our approach formulates advantage estimation in policy gradient (PG) methods as pairwise preference classification, enabling principled uncertainty quantification to guide routing. Uncertain pairs are forwarded to the LLM judge, while confident ones are evaluated by the RM. Experiments on RM benchmarks demonstrate that our uncertainty-based routing strategy significantly outperforms random judge calling at the same cost, and downstream alignment results showcase its effectiveness in improving online RLHF.

Ask a Strong LLM Judge when Your Reward Model is Uncertain
Zhenghao Xul
Qin Lu
Qingru Zhang
Liang Qiu
Ilgee Hong
Changlong Yu²
Wenlin Yao²
Yao Liu²
Haoming Jiang"
Lihong Li²
Hyokun Yun?
Tuo Zhao
'Georgia Institute of Technology
Amazon
Abstract
Reward model (RM) plays a pivotal role in reinforcement learning with human feedback (RLHF) for aligning large language models (LLMs). However, classical RMs trained on human preferences are vulnerable to reward hacking and generalize poorly to citoo--distributinn (OOD) inputs. By contrast, strong LLM judges equipped with reasoning capabilities demonstrate superior generalization, even without additional training, but incur significantly higher inference costs, limiting their applicability in online RLHF. In this work, we propose an uncertainty-basee routing framework that efficiently complements fast RM with strong but costly LLM judge. Our approach formulates advantage estimation in policy gradient (PG) methods pairwise preference classification, enabling principled uncertainty quantification to guide routing. Uncertain pairs are forwarded to the LLM judge, while confident ones are evaluated by the RM. Experiments RM benchmarks demonstrate that our uncertainty- bbased routing strategy significantly outperforms random judge calling the same cost, and downstream alignment results showcase its effectiveness in improving online RLHF. Our code is available at https://
gi github..c com www./neeggaoouuuaaaaooaaa hwwprggaaxuuggtcccchaae sanechh/uncertaity--oooo rrottr
1 Introduction
Reinforcement learning with human feedback (RLHF) is predominant approach for large language model (LLM) alignment and has shown great success improving the capabilities LLMs 63 This approach formulates alignment as RL problem in which the LLM, the actor, is tuned to maximize a reward function that reflects human preference. This RL problem is then solved by policy gradient (PG) type of methods, including PPO [39, GRPO
44
38 3
43 and RLOO [
The reward function in RLHF is typically realized by reward model (RM) learned from human- annotated preference data, assuming that human preference follows the Bradley-Terry (BT) model [5) This pointwise RM assigns scalar reward score r(æ: y) to a response y measuring the quality of this response to the prompt æ, which estimates the ground truth reward underlying the BT model 44]. A variant of this pointwise RM is the pairwise RM, or preference model (PM), which relax the BT assumption and assigns scalar preference score p(æ, yı, U2) pair of responses y and y2 given prompt æ, measuring the preference strength between the two responses 36 46 55 6T]. Both RMs typically trained based and directly outputting a scalar reward/prefference score. They are moderately fast and have been successfully integrated into the RLHF pipeline.
pretrained LLM, concatenating the input single sequence
"ork done during internship at Amazon. Emails: (zhenghaoxu, (hlanghaoxu.tourzhaa))aaae tourzhao) edu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

1.0
1.40
1.40
0.9
1.35
Uncertainty
Uncertainty
Accuracy
1.30
ID (Train]
0.6
1.25
(Val)
1.25
0.5
RM corr=-0.23 0.23
00D (RM)
0.4
1.20
1.20
1.25
1.30
1.35
1.40
000
000
Uncertainty
Reward Difference
(RB) (RM)
(Val)
(Train)
Figure 1: Left: Average accuracy of pairwise RM uncertainty quantiles on RewardBench (RB) and RM-Bench (RM). Each scatter point corresponds to 10% of the data corresponding to quantile. The accuracy shows negative Spearman's rank correlation with uncertainty, with p-values less than 10-29 and 10
10-136 136 on RewardBench and RM-Bench, respectively, suggesting that uncertain pairs are more likely get wrong rankings. Middle: Uncertainty gap between ID SaapStterr--refereneee train and validation sets) and OOD datasets (RewardBench and RM--ench), where the uncertainty scores given by our RM range from Uncertainty scores are averaged within 10 bins divided by reward difference. The marker size indicates the portion of data contained within the bin, and the shaded area represents the range within standard deviation. Right: Overall averaged uncertainty scores. The uncertainty scores are consistently higher on OOD data.
However, RM could be vulnerable to hacking 2 and susceptible spurious features such specific styles [15]. For example, when tested on the hard subset of RM-Bench [3I, which evaluates the RM's ability distinguish subtle content changes and resistance style biases, even the state-of-the-an models like Wewokk-Reward-Llarmee 1.llmm3..1-30 1-B (27 struggle to achieve a higher accuracy (46.6%) than random guess (50%). Because the RM is trained rather limited human preference data, which cannot exhaustively cover all possible responses, the RM possesses a lot of epistemic uncertainty and falls short in making reliable predictions when facing -iit--ddittibtttoon (OOD) data. As illustrated in Figure _ the RM accuracy can drop significantly on uncertain OOD data. Therefore, RM is still far from satisfactory objective.
Given the issue of standard RMM, recent works turn to strong generative LLM judges for more reliable reward and preference annotations [62]. The LLM judge concatenates the input sequence with judge rubrics and autoregressively generates an output sequence containing the verdict, which can be extracted by simple pattern matching. By leveraging long chain-of-though (CoT), strong LLM judge can reason before giving a final answer, enabling inference time scaling for a more reliable return [53 34 58 18. For example, Deepseek-RI (18) achieve 78.9% accuracy the hard
subset of RM-Bench (see Table outperforming traditional RMs by huge amount.
Although LLM judges make more accurate predictions, they significantly more costly than RMs due to their reliance on autoregressive generation and long COT. Consequently, their inference take many times longer than that of standard scalar RMs, even with ample hardware and parallel execution. This high latency renders them a bottleneck in policy optimization, making their direct deployment in online RLHF pipelines intractable.
To address all these challenging issues of RM and LLM-as-a-judge,, we propose an uncertainty- routing framework to provide reliable reward signals at an affordable cost. We first quantify the uncertainty of RM predictions, and then use the uncertainty score as an indicator for routing. the uncertainty is above a threshold, we recognize the data an OOD sample and send it to the strong LLM judge for a more accurate verdict. If the uncertainty is low, then the sample is more likely to be in-distribution (ID), where the RM can provide a confident prediction at a fast speed, no autoregressive decoding is required. Applying this uncertainty-but -based routing, we can complement standard RM with strong LLM judge to improve OOD performance with lower cost, striking balance between the two, and making it capable of enhancing the downstream online RLHE.
In particular, use the pairwise PM instead of pointwise RM, because they can better capture human preference beyond the BT model [36 46 61.. Moreover, pointwise BT RM indefinite.
making difficult to quantify its uncertainty from human preference data (more details in Section 2.3

---

Contrarily, pairwise PM learning is well-defined classification problem, and thus various principled uncertainty quantification methods can be applied, and we particularly use SNGP [2] 29 since only requires single model and inference once for each pair. While PM does not directly serve as an RI objective, can be used to estimate the advantage and thus applied a class of PG methods, including GRPO [ and RLOO ①
We conduct experiments demonstrate the efficacy our uncertainty-base routing method. Firstly, we evaluate on RM benchmarks, showing that sending uncertain samples to strong LLM judge can improve preference prediction accuracy without incurring too much cost. Then, we compare the uncertainty router with randomly routing the same number of samples to the judge. We evaluate their accuracy on reward benchmarks and apply them to downstream alignment. As illustrated in Figure and detailed in Tables 2 and 6 routing the uncertain samples can bring more improvement compared to random routing, showcasing the efficacy using uncertainty as a routing indicator.
Uncertainty
92.3% RR
26.4
Uncertainty
Reward Uerch Accuracy
92-
Random
26.2
AlarrEval LC
25.0
89
25.6
25.4% (No Routing)
Routing!
Percentage Calls
Percentage Calls (S6)
Figure 2: Uncertainty-based routing outperforms random routing with the same number of LLM judge calls preference accuracy and downstream alignment. See Figure 3 for other benchmarks.
The paper is organized as follows: Section introduces the preliminaries, Section B introduces main methods, Section provides experiments on reward benchmarks and downstream alignment, and Section [ makes concluding remarks. Additional related work and experimental details provided in the appendix.
Preliminaries
2.1 Reinforcement Learning with Human Feedback
Consider an LLM with parameter O, which takes a length-Zin sequence æ [[],,.... Tin as input and outputs a length- response = [y/1..... M.n.l sampled from its conditional probability distribution T- x) over all possible output sequences. Reinforcement learning with human feedback (RLHF) aims to maximize the expected reward under the LLM policy πy. where the reward function is usually given by reward model (RM) learned from human preference data. The preference data Dpref = {xx, you yr)} are assumed to be following the Bradley-Tere (BT) model [ with some unknown ground truth reward function r" (:;
Ply > yı x)) o(r (x: yw) r" (x, y:)),
(1)
where yo > Yi denotes that response you is preferred over response yi for prompt æ. and a (xx) = 1/(1 e-*) denotes the sigmoid function. A pointwise RM r(,,-) can thus be trained with the following maximum likelihood estimation (MLE) loss:
min ApoiRRM( (r) == Be (1) log((a yw) r( y:))))
(m.
Given the RM, RLHF aims solve the following RL problem prompt dataset D:
max Jpolicy (0) = D.y-mool)) [r(:;, y)] - BE ( (x) m- (- [x)]]
where (- -) is reference policy, (- || is the Kullback-Leibee (KL) divergence between two probability distributions, and B > is the regularization factor. The problem (3) usually solved
via policy gradient (PG, [54] 45 methods, including PPO [39]. RLOO 1 and GRPO 43)
One can also train pairwise RM, also called preference model (PM), to estimate the ground truth reward difference, p(æ, (,, yı, y2) ss r* rx, (x, y1) r (x, y2), which is equivalent to estimating the logits
3

---

in binary classification. The PM can be trained with the following loss:
min (2 J-
fpaitte (p) T Mil lo (og(a yves 9()))
If the data come from the BT model, then ideally minimizing yields PM consistent with the ground truth reward difference, which is sufficient for constructing advantage estimates for RLOO and GRPO. We provide more details when introducing our methods in Section 3.2
2.2 LLM Judge
Powerful LLMs (e.g., GPT-4) are increasingly used as automated judges 4 16 62 to reduce costly human annotations. This AI feedback approach prompts an LLM judge to evaluate competing responses (y), y2) for an input æ, providing preference labels (y/w: yr) at scale. The quality LLM judges benefits significantly from sfain-of-thouglle (CoT) prompting [53] and recent progress in LLM reasoning capabilities, which incentivizes the LLM judge to generate a long reasoning path before giving verdict, enhancing the reliability and transparency of evaluations (30..
Despite being cheaper than human annotation, LL---as---judge still incurs significant computational cost due to the resource requirements for inference. Furthermore, high inference latency, especially when generating detailed reasoning. can slow down the process and present practical challenges to apply to online RLHF [51). This necessitates the balance of feedback quality, speed and cost.
2.3 Uncertainty Quantification
Uncertainty quantification (UQ) aims to provide calibrated estimates of confidence associated with model predictions. In deep supervised learning, various UQ methods have been developed, particularly for classification tasks, including MC Dropout [13, Deep Ensembles [24], and methods focusing on distance-awarenee in input or feature space, such as DUQ 47 and SNGP 28 29
In particular, SNGP combines spectral normalization with a Gaussian process layer for uncertainty estimation with distance awareness, which is useful for OOD detection without the multiple inference cost of MC dropouts or ensembles. For binary classification data, outputs logit that indicates the alcatoric uncertainty (irreducible due to the nature of data distribution), divided by variance factor measuring the distance to the training set, which indicates the epistemic uncertainty (reducible by expanding data coverage).
Recent research has explored uncertainty quantification methods for RMs, such as LoRA ensembles [59] and last layer embedding [60]. While these approaches attempt to estimate uncertainty for pointwise RMs to enhance reliability, they face fundamental limitations. A key challenge that pointwise RMs are inherently indefinite under the Bradley-Terry preference model adding prompt- dependent-only bias term yields the same preference distribution, making the UQ problem ill-defined. This ambiguity complicates the interpretation of uncertainty estimates from ensemble kernel/GP methods. For instance, high variance in RM ensemble predictions could indicate either genuine uncertainty or simply convergence of RMs to different but equivalent solutions. While high-quality scalar ratings of individual responses could make pointwise RM learning a regression problem where UQ is well-posed [T], such data is typically less available than human preference data [50] 52) In this work, we specifically address uncertainty quantification for pairwise RM (PM) trained on
human preference data that are more widely used in the RLHF literature
63
44
38
3 Method
We aim to address the issue of poor generalization of RM on OOD data in order to get better down- stream alignment performance. Instead of improving the RM itself, we investigate complementing the RM with strong external LLM judge, which provides more reliable preference feedback. However, such an external LLM judge incurs high inference cost and latency, making unrealistic, given the limited computational budget, to evaluate every response generated by the actor model during online RLHF Therefore, we need to specify a strategy to switch between cheap but weak RM and the strong but expensive L.M judge in order to maximize the gain from the limited number of judge calls. To achieve this, we propose a routing framework based on RM uncertainty quantification (UQ), sending the data that RM is uncertain about to the LLM judge for further evaluation.

---

3.1 Reward Difference Estimation with Uncertainty Quantification
We first train a pairwise RM (PM), "(""")), estimate the ground-truth reward difference with uncer- tainty quantification using preference data Pretf = {(x, Yee ). In particular, we concatenate the prompt and two responses into single input sequence using chat template (details in Appendix [A]. We take the hidden states at the last token of the concatenated input as an embedding vector and add head outputting the classification logit p(æ, Yur y/) used 44
To mitigate the position bias that the two responses are concatenated only the chosen-rejected order, we swap the positions and flip the labels, so that the augmented dataset DD Prref contains both (x. Yun: W1 1) and (æ. y) Yw. 0). We minimize the following classification loss get a PM:
p: ( .22 (p) = E (...
min
- log(o yı, (2)))
- (1 2) loglo y2, y2, y1)))
(5)
Given the well-posednes of the classification problem quantification methods to detect OOD data. In particular, we apply the spectral-normalized Gaussian process (SNGP, [28 (9) method, as it only requires single model and infers once for each pair. When applying SNGP to LLM-based PM, we add spectral normalization
can apply principled uncertainty
(35] to the linear output layer in transformer blocks, take the final hidden states at the last token h h(x, y1 y2) E R Ps and pass it to a Gaussian process (GP) layer (approximated by random features) to get the logit g(h) that corresponds to the reward difference:
(h) +(h) ((h) = 20 cos Wh + b)
(6)
where 3 E Dr, D, the number of random features, of the kernel amplitude, W E D.Dxx is fixed matrix with its entries i.i.d. sampled from standard Gaussian A/0,, 1). and Pr is fixed vector with its entries i.idd sampled from uniform distribution Unif 2..T...
During training, we plug p(a,, y1, y2) g(((x, yı: y2)) into (5) and apply gradient methods to update all hidden weights, except the fixed weights in the GP layer, i.e., W and b. After training completes, we add additional epoch compute the posterior covariance matrix
Σ = invE\E='),
Σ- = rI +
- (1.))11(())))
where p. ((x(()) Vi (8) During inference, we compute the reward difference p and uncertainty u as
()) and Φ. ((x(1)
(8)
U2
p(æ, y2) u(x, yı, y2) u(x, yı. y2) = /1+ (0))-(hh)))))
(
(8)
where g(h) and (hh) are defined in (6) and E is defined in 7 À is scaling factor.
In this SNGP-PM, the logit g quantifies the aleatoric uncertainty from the BT model, and the variance- induced uncertainty quantifies the epistemic uncertainty due to limited training data. The aleatoric uncertainty is not reducible as inherent in human preference; thus, applying an LLM judge to the aleatoric uncertain samples may not bring much improvement. On the other hand, epistemic uncertainty is reducible; thus, strong LLM judge with good generalization may help improve the prediction epistemic uncertain samples. Therefore,
uncertainty quantifier.
Remark As mentioned in Section 2.3 we consider PM instead of pointwise RM for preference data under the BT assumption because the uncertainty quantification problem is not well-posed for the latter. Consider a pointwise RM r(x,, y), it consistent with any other RM in the form of r(a, y) + s(x) under the BT model, and we cannot guarantee which one is returned by minimizing (2) even with infinite data. Therefore, is difficult to assign a prior distribution on the pointwise RM, which is crucial for uncertainty quantification. In contrast, PM is well defined within the data support and unique in the population sense. Therefore, one can measure the distance from the data to the support of the training set for epistemic uncertainty quantification.
5

---

3.2 Advantage Estimator from Reward Differences under Uncertainty-Basee Routing
When serving the PM, we set a threshold and use the uncertainty in ( to route to strong LLM judge: if the uncertainty is below the threshold, we directly use the prediction from the PM; if the uncertainty is above the threshold, we call the LLM judge and use its prediction in turn. The estimated pairwise reward differences are then used to construct the advantage values, enabling downstream RLHF with a class of policy gradient (PG) methods, including GRPO [8] and RLOO M.
More precisely, the (stochastic) policy gradient is computed by taking the gradient of policy loss:
B Σ
policc (0) =
A, log no (y: a2);
where { B is a batch of prompts, yi ~ (T x.), and A; = r(æ;; yi) b(x;) is the estimated
advantage of response y; conditioned on prompt æ; compared to a baseline b(x;;) that only depends
on the prompt z;, To reduce the variance, the baseline is usually set as the value function b(z) = By--ro(-Im)] [r(z,, y)]. which is approximated by critic model in PPO [39]. or by Monte Carlo (MC) samples as in RLOO []] and GRPO [43] For simplicity, we consider RLOO, which generates group of responses for each prompt and uses the leave-one-out average estimate the baseline and
K advantages. Suppose {yr} j=1 are K responses generated from ( a.), then the advantage for this group corresponding to prompt is given by
A: = r(x, y:) K
Σ ((x,y))) jEH
(r((, - r(x,, yz)).
(10)
K K-1
In view of (10), estimating the advantage only requires reward differences between responses within each group, and thus our SNGP-PM with an uncertainty router is applicable. When the SNGP-PM is certain about the comparison, we use its predicted reward difference directly. When the SNGP-PM is uncertain, we call strong LLM judge, assuming can produce reliable feedback on the comparison.
We restrict the return to be one of the three labels, indicating that y, is better, Yj is better, or they are tied (see Appendix A for details). Given the working assumption that the external judge is strong, we assign a high confidence score (corresponding to near or 0 probability) in case it predicts that one of the responses is better, and assign low confidence score (corresponding case it predicts tied which indicates the occurrence of aleatoric uncertainty. The reward differences corresponding the confidence scores can be obtained by applying the inverse of the sigmoid function. Combining the two sources of reward difference via the uncertainty router, our serving PM makes the following prediction on tuple (æ,, yi. y5):
1/2 probability) in
o (1-6),
yi > Yj J
p(a; yi my) p(ar, yi Vj): u(x yt, yj) S u,
(xz yo. HJ), u(x y:, y,) > T.
((.
(c).
yi - yj-
(1/2),
y L
Yj-
where ū is the routing threshold, 0 <<< < 1, and - -, d denote the verdicts from the judge.
Using the reward differences, we compute the advantage estimate (TOp and plug it into @ Then, adding the KL regularization yields the policy loss associated with the prompt x for an RLOO step.
K
LRLOO = K(K - 1)
p(x. yi, yı) log To (y: x)
1=1 jt1
+ +DDKL(TT( x) Tree(- x)).
(11)
The downstream RLHF is then performed by iteratively sampling a batch of prompts, generating responses from the policy model, and updating the weights by taking gradient step on the loss (11).
Remark 2
The PM-based RLOO loss II has a connection with Nash learning with human feedback
611. These works formulate the alignment problem as minimax game instead of an RL problem 33. In this work, we still follow the RL framework and
or self-play RLHF (36 46 57
42
6
use PM only to construct advantage estimates. Our work focuses on efficiently complementing the PM with LLM-as-a-jine -Maass--juddee instead of improving the downstream alignment method itself.

---

Experiments
In this section, we present our experiments that examine the performance of SNGP-PM with an uncertainty router to the LLM judge and its benefit to downstream alignment.
4.1 Experiment Setup
Models. For both pairwise RM (PM) training and downstream alignment, we use Llama-3. 1-8B- Instruct [17] as our base model. When serving PM with an uncertainty router, we use the DeepSeek-Re R1 18 model as a judge, as already achieves high accuracy on reward benchmarks without specific fine-tuning (see Tables 2 and
Datasets. For PM training, we use the SeepSteer2-PPeffreete dataset [51]. which consists of 7,118 high-quality preference pairs with 6,766 training data pairs and 352 validation data pairs. For downstream alignment, we use subset (the first 33%) of the prompt from the Ultrafeedback dataset [8. which consists of about 20k prompts covering various domains including instruction following, truthfulness, honesty, and helpfulness.
Benchmarks. For RM evaluation, we use RewardBench 25 and RM-Beenh [31 datasets. Re- wardBench contains 2,985 preference pairs measuring the RM's capabilities over the categories of chat, chat hard, safety, and reasoning. RM-BBechh contains 1, 1,327 prompts, each associated with chosen responses and 3 rejected responses, consisting of 11,943 pairwise comparisons in total. The responses in RM-Bench are constructed to amplify the style bias, making a hard benchmark for RM to accurately make correct predictions. We report the accuracy of distinguishing chosen and rejected responses.
For downstream aligned policy models, evaluate their performance three widely adopted open- ended instruction following benchmarks: Arena-Hard-wy [26), AlpacaEval 2.0 [12]. and MT-Bench 662.. These benchmarks ask the model to generate answers to wide range of open-ended questions and use strong judge models to assess the quality of the response. We follow each benchmark's evaluation protocol and report corresponding scores. For Arena-Hard-vOk.
report the win rate (WR). For AlpacaEval 2.0, we report the WR and length-controlled (LCC) WR. For MT--Bench, we report the scores on two turns and their average. More details are provided in Appendix
Baseline. To show that SNGP uncertainty quantification would not affect the accuracy, we train a standard PM without the GP head as baseline. To validate the efficacy of our uncertaint--bssed routing approach, we experiment with different uncertainty thresholds and compare with random routing. When random routing is applied, still use the same uncertainty threshold, but only for counting the number of required calls within the batch. We then randomly sample the indices and call DeepSeek- R1 on those pairs.
4.2 Uncertainty-basee Routing Improves OOD Generalization
We train a PM with SNGP as specified in Section B.Iee on Llama-3. --nt-tntttttttt and HelpSteer2- Preference dataset. We augment the data by swapping the two responses for each prompt and apply the message format in Appendix A to construct the actual dataset used for PM training. The TelpsSterr-Prrefeee
dataset contains preference strength (1,2,3} associated with each tuple, so we use the following scaled BT loss as suggested [51]:
Lecaated (p) == - -E (a..)) (...,).......))) .)) [s
(p((x, y1, U2))))) + s. (1 -1-2 2) ()) log(a y2, (1)))))))) ))))
We train for 2 epochs to prevent overfitting, and use the third epoch to compute the covariance matrix used for SNGP uncertainty estimation, during which the weights are frozen. For the baseline, we replace the GP layer with simple linear head and train standard PM using the same data and loss function for epochs. More details are provided in Appendix C.I
The standard PM and SNGP-PM are evaluated on the HelpSteer2-Preferee Prefereeneeeeeeeee validation set, Reward- Bench and RM-Bench, and the results are provided in Table [ As illustrated, the models perform comparably with less than 1% overall accuracy difference, suggesting that the additional uncertainty quantification component does not introduce significant overhead prediction accuracy.
We then evaluate our uncertainty routing strategy on RewardBench and RM-Bench. To mitigate position bias during inference, for each tuple (æ, Vw: y1) of prompt, chosen and rejected responses, we use p(m. pp2.,a [ww) as the predicted reward difference and "0 u as the un- certainty. We set the threshold in (10.0, 1.45, 1.40, 1.35, 1.30). ). and send the tuples whose uncertainty
2
7

---

Table 1: Comparison of standard preference model and SNGP-PM on HelpSteer2-Preefreee valida- tion set, Reward Bench, and RM-Bench The performance of two models are comparable.
Validation
Reward Bench
RM Bench
Model
avg-
chal
chat hurd
safety
reasoming
arg.
chat
math
cude
salitty
0.877
0.731 0.738
0.894
0.918
SNGP-PM PM
0,793
0.873
0.958
0.894
0,900
0.680
0.671
0.595
0.542
0.912
is beyond the threshold to the DeepSeek-RI judge using the template specified in Appendix A The sign of the final preference indicates the correctness of the prediction. For the baseline, we choose random routing that routes exactly the same number of tuples to DeepSeek-RI but in random way.
The prediction accuracies RewardBench and RM-Bench are presented in Tables 2 and re- spectively. From the tables, it is shown that calling strong LLM judges improves the RM accuracy, especially on the hard domains where PM (no routing) performs poorly, such chat hard and reasoning sections in RewardBench, and math and coding in RM Bench. Moreover, the threshold routing approach is significantly more efficient than random routing, achieving higher accuracy gains with the same amount of total judge calls.
Table 2: Performance comparison on RewardBench with different routing strategies. The thresholds chosen in (10.0, 1.45, 1.40, 1.35, 1.30).
Reward Bench (%)
Routing
Num of Calls
chat
chat hard
safety
reasoning
avg. (vs rand)
No routing
95.8
73.8
89.4
90.0
87.3
58 (1.9%)
96.1
74.8
89.5
91.7
88.0 (+0.8)
274 (9.2%)
96.4
76.8
89.8
93.7
89.2 (+1.7)
Uncertainty
719 (24.1
96.9
80.3
89.8
95.4
90.6 (+2.4)
1270 (42.5%)
98.3
81.2
90.0
97.0
91.6 (+2.5)
58 (1.9%)
95.5
74.0
89.4
89.9
87.2
274 (9.2%)
96.4
73.7
89.5
90.4
87.5
Random
719 (24.1% ())
95.0
75.9
90.2
91.5
88.2
1270 (42.5%)
95.5
77.5
91.6
91.9
89.1
DeepSeek-RI R1 100%
95.5
85.8
91.1
96.9
92.3
Table 3: Computational costs for different routing strategies on RewardBench.
Uncertainty Threshold
10
1.45
1.4
1.35
1.3
<|
Num Calls (Ratio) Inference Time (s) - Uncertainty
0%
1.9%
9.2%
24.1%
42.5%
100%
107
156
245
540
609
1113
Inference Time (s) Random
107
149
208
361
541
1113
Since hard instances may take more inference time from the LLM judge, we further record the wall clock time running the evaluations and compare the performance of uncertainty-based and random routing with the same amount of inference time. run evaluations on NVIDIA-A 100 GPUs in parallel, each processing 25% of the comparisons with an SNGP-PM. We then gather all routed instances and send them to the remote-hosted DeepSeek-RI judge parallel, which can process 200 requests per minute. As shown in Tables and S uncertainty -based routing requires more inference time than random routing, suggesting that the uncertain instances are indeed harder. Nevertheless, the uncertainty-basee based routing strategy achieves higher accuracy with less time. These experiments validate the efficacy of our uncertainty-but based routing approach.
4.3 Uncertainty- -basd Routing Improves Downstream Alignment
We then experiment on downstream alignment. We apply RLOO (11) for online RLHE For each group of K K-responses to same prompt, we construct preference matrix P TR. KxX to estimate the advantages. We send K K(K 1) ordered pairs to SNGP-PM and get P..j p(æ, yi y,) and

---

Table 4: Performance comparison on RM-BBench with different routing strategies. The thresholds are chosen {10.0, 1.45, 1.40, 1.35, 1.30).
RM Bench (%)
Routing
Num of Calls
chat
math
code
safety
easy
normal
hard
avg. (vs rand)
No routing
0
67.1
59.5
54.2
91.2
87.2
72.0
44.9
68.0
242 (2.0%)
68.7
60.0
54.7
91.4
87.4
73.3
45.5
68.7 (+0.2)
1285 (10.7%)
69.6
64.9
59.7
92.0
89.1
76.3
49.2
71.6 (+1.6)
Uncertainty
3188 (26.7%)
71.3
73.8
68.7
92.6
91.4
81.5
56.9
76.6 (+3.1)
5270 (44.1%) %))
73.2
83.5
78.3
92.7
93.6
86.9
65.3
81.9 (+4.8)
(2.0%)
67.5
60.2
54.9
91.2
87.4
72.4
45.6
68.5
1285 (10.7%)
68.0
63.6
57.2
91.3
88.0
74.1
48.0
70.0
Random
3188 (6.7%)
69.6
69.3
63.6
91.3
89.4
77.0
54.0
73.5
5270 (44.1% ())
70.1
76.8
69.9
91.6
91.0
81.3
59.1
77.1
DeepSeck-Ri R1
100%
76.8
95.7
87.8
92.0
94.0
91.3
78.9
88.1
Table 5: Computational costs for different routing strategies RM-Bench.
Trigger Threshold
10
1.45
1.4
1.35
1.3
<1
Num of Calls (Ratio) Inference Time (s) Uncertainty
0%
2.0%
10.7%
26.7%
44.1%
100%
518
632
1113
2200
3007
5642
Inference Time (8) Random
518
625
1093
1979
2615
5642
ye y;) We let P P-PT and U + U+UT matrix and symmetric uncertainty matrix. We then follow Section 3.2 to get feedback from the DeepSeek-RI judge when uncertainty is above the threshold and compute the advantages accordingly.
2 to enforce an anti-symmeetrc preference
Given resource constraints, we set K = 4 and train on the first 33% of the Ulrrfeedback prompts epoch. We set the threshold in { {10.0, 1.35, 1.30, 1.20} and compare the uncertainty- and random routers. More training and evaluation details are provided in Appendices C.2 and Ck As shown Table 6. complementing the PM with DeepSeek-R RI as judge during RLHF brings improvement to downstream policy performance with small portion of calls. Moreover, uncertainty- based routing general exhibits higher improvement, showcasing the efficacy our routing strategy.
Table 6: Performance of downstream models trained with different routing strategies and thresholds. For random routing, we use the same threshold for counting but send samples to the judge randomly.
Arena-Hard (%)
AlpacaEval (%)
MT-Bench
Model
Num of Calls
WR
WR
Turn
Turn 2
Avg
Base model
24.5
22.31
23.63
7.98
6.80
7.47
No routing (10.0)
28.1
25.40
27.35
8.19
6.98
7.65
Uncertainty (1.35)
7668 (6.6%)
28.9
26.28
28.97
8.05
7.19
7.65
Uncertainty (1.30)
10522 (9.0%)
28.9
26.34
28.53
8.03
7.13
7.63
Uncertainty (1.20)
21363 (18.3%)
29.8
26.45
28.91
7.40
7.71
Random (1.35)
7523 (6.4%)
26.5
25.70
28.55
7.20
7.71
Random (1.30)
10854 (9.3%)
27.7
25.14
28.29
7.93
6.74
7.41
Random (1.20)
20474 (17.5%)
28.5
25.98
28.51
8.00
6.62
7.45
5 Additional Related Work
Pointwise and pairwise reward model. Classical pointwise reward models (RM) are typically
learned from human preference data under the Bradley-Terry (BT) model (5), which consists of
main component in RLHF 7
63
44
38) These methods train the pointwise RM by performing
maximum likelihood estimation (MLE) on the preference data and could lead to RMs that lack
calibration across different prompts [56] When absolute rating data is available, where there

---

attribute scores assigned to individual responses, the pointwise RM can also be trained via regression However, high-quality rating data less abundant than preference data, as fine- fine-grained scalar scores typically require more human labor and precise rating rubries for reliable annotations
T0] 49
Recently, some works have relaxed the BT assumption and explored using pairwise RM, also called preference model (PM), to model general human preference [61]. Such a PM is trained by classification on human preference data and can serve as an evaluator or preference annotator [22] 32 It can also serve as the alignment objective in Nash learning with human feedback [36 46 57 42
23
50 52
or self-play policy optimization [55 61 [6]. While formulated differently, these alignment methods
have deep connection with standard policy gradient methods with baseline estimated via group averaged reward [48], including RLOO [ and GRPO (3).
routing framework is conceptually aligned with active learning, which seeks to improve data efficiency by strategically selecting the most informative instances for labeling by an oracle [4). In our work, the strong L.L. judge acts as the oracle, and our uncertainty score serves as the acquisition function to identify the most uncertain pairs of responses. Classic query strategies in active learning often rely model uncertainty, such as selecting the least confident predictions or using a committee of models to find contentious examples [11. which have been extended to deep learning through Bayesian methods [4). While active learning has been explored for learning reward/preference functions from limited interaction in the RL context [9] our primary contribution is the application of this principle to direct online RLHF setting. Here, the feedback from the oracle is used immediately to construct advantage estimates for policy gradient updates, and feeding these newly labeled high-uncertainty pairs back into the reward model for continued training is a natural extension that would constitute a full active learning cycle.
Connection to active learning. Our uncertainty-
Conclusion
In this work, we propose an uncertainty-based routing strategy complement cheap but poorly generalized RM with more reliable but expensive LLM-as-a-judge efficiently for RLHF. Experiments demonstrate improvement on reward benchmarks and downstream alignment, and our uncertainty- based routing strategy outperforms random routing, showcasing the efficacy of our method, which is measured by the number of judge calls. While SNGP does not significantly increase the computational cost, a future analysis using iso-flops could provide a more fine-grained measure of computational cost by also accounting for the UQ method's 's overhead.
The current work directly uses DeepSeck-RR RI as judge, which is not specifically fine-tuned for the judge task and requires huge computational resources to host. One can potentially replace it with a smaller-scale generative RM [34] to further improve the judge quality and inference efficiency. Another future direction could be quantifying the hardness of the sample and the uncertainty of the LLM judge and enabling a hierarchical routing strategy, potentially unifying the RM under the generative paradigm and allocating inference time budget more efficiently. A related avenue is exploring alternative UQ methods to optimize the trade-off between uncertainty quality and the routing framework's 's overhead. Besides L.M judges, the uncertainty- most uncertain samples to human annotators, potentially augmenting the human preference dataset with previously not covered responses from the policy model. This could help close the gap between reward training and actual policy model distributions.
router can also send the
Acknowledgments and Disclosure of Funding
We thank Rongzhi Zhang for the discussions at the early phase of our work. We thank the anonymous reviewers for their constructive feedback and suggestions.
References
[1] A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv: 14740, 2024.
[2] D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in ai safety. arXiv preprint wwXiv:1606.05550 2016. 2016.
10

---

[3] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint (rXXiv2204.0586) 2022.
[4] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirho- seini, C. McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08077 2022.
[5] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
[6] D. Calandriello, Guo, R. Munos, M. Rowland, Y. Tang, B. A. Pires, P. H. Richemond, C. L. Lan, M. Valko, T. Liu, et al. Human alignment of large language models through online preference optimisation. arXiv preprint arXiv:2403.0866. 2024.
[7] Christiano, Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[8] G. Cui, L. Yuan, N. Ding, G. Yao, B. He, W. Zhu, Ni, G. Xie, R. Xie, Lin, et al. Ultrafeed- back: Boosting language models with scaled ai feedback. arXiv preprint arXiv:2310.01377, 2023.
[9] C. Daniel, M. Viering, J. Metz, O. Kroemer, and J. Peters. Active reward learning. Roboties: Science and systems, volume 98, 2014.
[10] Y. Dong, 7. Wang, M. N. Sreedhar, X. Wu, and O. Kuchaiev. Steerlm: Attribute conditioned sft as an (user-steerable) alternative to rlhf. arXiv preprint arX1v:2310.05344, 2023.
[11] N. Dorka. Quantile regression for distributional reward models in rlbf. arXiv preprint arXiv:2409. 10164, 2024.
[12] Y. Dubois, B. Galambosi, P. Liang. and T. B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint S.XXi::2404.044777 2024.
[13] Y. Gal and Z. Ghahramani. Dropout as bayesian approximation: Representing model uncer- tainty in deep learning. In international conference on machine learning, pages 1050-1059. PMLR, 2016.
[14) Y. Gal, R. Islam, and Ghahramani. Deep bayesian active learning with image data. In International conference machine learning, pages 11 1183-1192. PMLR, 2017.
[15] R. Geirhos, J. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning deep neural networks. Nature Machine Intelligence, 2( 1)665-677, 2020.
[16] Gilardi, M. Alizadeh, and M. Kubli. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e2335016111 2023.
[17] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama herd of models. arXiv preprint arXiv:2407.21783, 2024.
[18] D. Guo, D. Yang, H. Zhang, J. Song. R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
[19] N. Houlsby, F. Huszár, Z. Ghahramani, and M. Lengyel. Bayesian active learning for classifica- tion and preference learning. arXiv preprint arXiv: I 112.5745, 2011.
[20] Hu, X. Wu, 7. Zhu, Xianyu, W. Wang, Zhang, and Y. Cao. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.11143, 2024.
[21] H. Ivison, Y. Wang. J. Liu, Z. Wu, V. Pyatkin, N. Lambert, N. A. Smith, Y. Choi, and H. Ha- jishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. Advances in neural information processing systems, 37:36602-3666 2024.
11

---

[22] D. Jiang, X. Ren, and B. Y. Lin. Lim-blender, Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint wrXiv:2366022661, 2023.
[23] A. Köpf. Y. Kilcher, D. Von Rütte, S. Anagnostidis, Z. R. Tam, K. Stevens, A. Barhoum, D. Nguyen, O. Stanley, R. Nagyfi, et al. Openassistant conversations-demorrattii large language model alignment. Advances in Neural Information Processing Systems, 36:47669- 47681, 2023.
[24] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
[25] N. Lambert, V. Pyatkin, J. Morrison, L. Miranda, B. Y. Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y. Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint trXiv:2403.13787 2024.
[26] T. Li, W.-L. Chiang, E. Frick, L. Dunlap, T. Wu, B. Zhu, J. E. Gonzalez, and I. Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint ar arviv:2406. 1939, 2024.
[27] C. Y. Liu, L. Zeng, J. Liu, R. Yan, J. He, C. Wang, S. Yan, Y. Liu, and Y. Zhou. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410. 1485514100.8511,, 2024.
[28] J. Liu, Z. Lin, S. Padhy, D. Tran, T. Bedrax Weiss, and B. Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in neural information processing systems, 33:7498-7512 2020.
[29] J. Z. Liu, S. Padhy, J. Ren, Z. Lin, Y. Wen, G. Jerfel, Z. Nado, J. Snoek, D. Tran, and B. Lakshminarayanan. A simple approach to improve single-model deep uncertainty via distance- awareness. Journal of Machine Learning Research, 24(42): -6, 2023.
[30] Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu. G-eval: Nig evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 25 2511-2522, 2023.
[31] Y. Liu, Z. Yao, R. Min, Y. Cao, L. Hou, and J. Li. Rm-bench: Benchmarking reward models of language models with subtlety and style. arXiv preprint 2Xiv:2410.16184, 2024.
[32] Y. Liu, H. Zhou, Z. Guo, E. Shareghi, I. Vulić, A. Korhonen, and N. Collier. Aligning with human judgement: The role of pairwise preference in large language model evaluators. In First Conference on Language Modeling, 2024.
[33] X. Lou, D. Yan, W. Shen, Y. Yan, J. Xie, and J. Zhang. Uncertainty-awarr reward model: Teaching reward models to know what is unknown. arXiv preprint arXiv:2410.0088 2024.
[34] D. Mahan, D. Van Phung, R. Rafailov, C. Blagden, N. Lile, L. Castricato, J.PP. Fränken, C. Finn, and A. Albalak. Generative reward models. arXiv preprint arXiv:2410.128:3 2024.
[35] Miyato, Kataoka, M. Koyama, and Yoshida. Spectral normalization for generative adversarial networks. In International Conference Learning Representations, 2018.
[36] R. Munos, M. Valko, D. Calandriello, M. G. Azar, M. Rowland, Z. D. Guo, Y. Tang, M. Geist, T. Mesnard, A. Michi, et al. Nash learning from human feedback. arXiv preprini arXiv:231..00886, 18, 2023.
[37] Z. Nado, N. Band, M. Collier, J. Djolonga, M. Dusenberry, S. Farquhar, A. Filos, M. Havasi, R. Jenatton, G. Jerfel, J. Liu, Z. Mariet, J. Nixon, S. Padhy, J. Ren, T. Rudner, Y. Wen, F. Wenzel, K. Murphy, D. Sculley, B. Lakshminarayanan, J. Snock, Y. Gal, and D. Tran. Uncertainty Baselines: Benchmarks for uncertainty & robustness in deep learning. arXiv preprint arXiv: 106.04015 2021.
[38] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 012227730-27444 2022.
12

---

[39] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, arXiv preprint arXiv: 177Xv:1707.06347,222 2017.
[40] B. Settles. Active learning literature survey, 2009. URL http:::://igggialllibbob... wisc
edu/1793/60660)
[41] H. Seung, M. Opper, and Sompolinsky. Query by committee. In Proceedings of the fifth annual workshop on Computational learning theory, pages 287-294, 1992.
[42] L. Shani, A. Rosenberg, A. Cassel, O. Lang, Calandriello, Zipori, H. Noga, O. Keller, B. Piot, I. Szpektor, et al. Multi-turn reinforcement learning from preference human feedback. arXiv preprint urXiv:2405. 1....1148 14655, 2024.
[43] Z. Shao, P. Wang. Q Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning open language models. arXiv preprint arXiv:2402.03330 033000 2024.
[44] N. Stiemnon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-3021, 2020.
[45] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforce- ment learning with function approximation. Advances in neural information processing systems, 12, 1999.
[46] G. Swamy, C. Dann, R. Kidambi, Z. S. Wu, and A. Agarwal. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint arXiv:2401.0444 04006, 2024.
[47] J. Van Amersfoort, L. Smith, Y. W. Teh, and Y. Gal. Uncertainty estimation using a single deep deterministic neural network. In International conference on machine learning, pages 9690-9700. PMLR, 2020.
[48] M. Vojnovic and S.-Y. Yun. What is the alignment objective of grpo? arXiv preprint arXiv:2502. 18548, 2025.
[49] H. Wang. W. Xiong. T. Xie, H. Zhao, and T. Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts arXiv preprint arXiv:2406. /2845, 2024.
[50] Z. Wang, Y. Dong, J. Zeng, V. Adams, M. N. Sreedhar, D. Egert, O. Delalleau, J. P. Scowcroft, N. Kant, A. Swope, et al. Helpsteer: Multi-attribute helpfulness dataset for steerlm. arXiv preprint arXiv:2311.09522 2023.
[51] Z. Wang, A. Bukharin, O. Delalleau, D. Egert, G. Shen, J. Zeng, O. Kuchaiev, and Y. Dong. helpsteer2-preferennee Complementing ratings with preferences. arXiv preprins arXiv:2410.01257, 2024.
[52] Z. Wang. Y. Dong, O. Delallcau, Zeng, G. Shen, D. Egert, J. Zhang, M. N. Sreedhar, and O. Kuchaiev. Helpsteer2: Open-source dataset for training nop-performing reward models. arXiv preprint arXiv:2406.08673, 2024.
[53] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. D. Zhou, et al. Chain-of- thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
[54] R. J. Williams. Simple statistical gradient- following algorithms for connectionist reinforcementt learning. Machine learning, 8:229-256, 1992.
[55] Y. Wu, Z. Sun, H. Yuan, K. Ji, Y. Yang. and Q Gu. Self-play preference optimization for language model alignment. arXiv preprint arXivv2405.00675 2024.
[56] W. Xu, X. Zuo, C. Xin, Y. Yue, L. Yan, and Y. Wu. A unified pairwise framework for rlhf: Bridging generative reward modeling and policy optimization. arXiv preprint 0445552504.04950, 2025.
13

---

[57] C. Ye, W. Xiong, Y. Zhang, N. Jiang, and T. Zhang. A theoretical analysis of nash learning from human feedback under general kl-regularized preference. arXiv e-prints, pages arXiv-2402. 2024.
[58] Y. Yu, Z. Chen, A. Zhang, L. Tan, C. Zhu, R. Y. Pang, Y. Qian, X. Wang, S. Gururangan, C. Zhang, et al. Sele-generated critiques boost reward modeling for language models. arXiv preprint arXiv:2411.16646, 2024.
[59] Y. Zhai, H. Zhang, Y. Lei, Y. Yu, K. Xu, D. Feng, B. Ding, and H. Wang. eepertaintt-pemaalizee reinforcement learning from human feedback with diverse reward lora ensembles. arXiv preprint arXiv:2401.00243, 2023.
[60] X. Zhang, J.-F. Ton, W. Shen, H. Wang, and Y. Liu. Mitigating reward overoptimization via lightweight uncertainty estimation. Advances in Neural Information Processing Systems, 37: 8 81717-81747, 2024.
[61] Y. Zhang, G. Zhang. Y. Wu, K. Xu, and Q Gu. Beyond bradley-terry models: general preference model for language model alignment. In ICLR 2025 Workshop Bidirectional Human-Al Alignment, 2025.
[62] L. Zheng, W.-L. Chiang, Y. Sheng. S. Zhuang, Z. Wu, Zhuang. Z. Lin, Z. Li, D. Li, E. Xing, al. Judging llm-as-a-judge with mt-bench and chathot arena. Advances in Neural Information Processing Systems, 36:46595 46623, 2023.
[63] D. M. Ziegler, N. Stiennon, J. Wu, B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irv- ing. Fine-tuning language models from human preferences. arXiv preprint arXiv: 08593, 2019.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims about RM and Luddee are supported by reward evaluation benchmarks in Section g The claim that our method can identify OOD data and high
uncertainty relates to low accuracy is supported by Figure T The claims about the proposed uncertainty based routing strategy are supported by reward experiments and downstream alignment experiments. They correctly reflect the paper's ss contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitation of using preference data is discussed in Section 2.3 the limita- tion of using a costly and not fine-tuned LLM judge is discussed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This is not theory paper and does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The methodology is introduced in Section with basic experiment setups in Section 4 Implementation details are provided in Appendices Aland and
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: All the datasets and base models are open-soureed and publicly available. The experiments are based open libraries such OpenRLHE. We share code at https:///itthu: com feomm/henghaouuueoo gwteecatttccceee banteechunccerainty--o. routerr
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.co tttpcc//pbbbc cuublc//guidee wttpg//ddeeeeeeuubmmmimceee www.//eoddesissiiiooaa wommissssoooiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide experiment setups in Section 4 and more details in Appendices A and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We have error bars and confidence intervals in, e.g., Figure 1 and Table 10
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
We provide compute resources detailed in Section and Appendix C (e.g.. NVIDIA-A100-80C GPUs).
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htt.p:/ http///wwww.p.c.......c.cccccaaa
Answer: [Yes]
Justification:
The research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper focuses on foundational research. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not have such kind of risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The creators of assets used in the paper are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
The paper uses LLM-as-a-judge to complement the reward models and the method is described in Sections 3 and 4 and Appendices. The paper also uses LLM-as-a- judge for downstream evaluations (Arena-Hard, AlpacaEval, MT-Bench)..
Justification:
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy http:://eerrrps..c//offeeeeeeeee for what should or should be described.
21

---

A Chat Templates
We use the messages format and apply the default chat template of to construct concatenated input for pairwise RM. For any prompt, response and response 2, the messages are constructed follows.
Messages for pairwise comparison
messages
"content"; {prompt), "role": "user",
"content": {response_1); "role": "assistant_1",
"content"; (response_2), "role"; "assistant_2",
]
the context has multiple turns, we append the messages from assistant_] and assistant_2 after the last message from user.
We the following template the prompt to request feedback from the LLM judge, which partially uses the HelpSteer2 rubrics for human annotation 52
http:///uullggdooo.o//ssssssstte -aaaaaal-aaaaaaaaaaaaaaaa Llama 8-8B- h ionjig.json
22

---

Prompt for LLM judge
are an impartial judge, tasked with evaluating the quality the two assistants' responses to the context displayed below. Your evaluation should be based on the following six criteria:
Helpfulness: Overall helpfulness of response the user's question instruction.
Correctness: Inclusion of all pertinent facts without errors.
Coherence: Consistency and clarity of expression.
Complexity: Intellectual depth required to write response (i.e., whether the response can written by anyone with basic language competency or requires deep domain expertise).
Verbosity: Amount of detail included in the reaponse, relative to what is asked for in the context.
Safety: Whether the response is free of any kind harmful, toxic, illegal content.
After carefully considering these criteria, determine which assistant's response is superior.
Output your final verdict by strictly following this format:
<label>1</label> if assistant A is better, <label><</label> if assistant is better, and <label>0</label> only if you really cannot tell their difference.
[The Start of Context]
(prompt)
(The End of Context.]
[The Start of Assistant A's Response] fresponse_1]
[The Assistant A's Response]
[The Start of Assistant B's Response] (response__)
[The End of Assistant B's Response]
B Policy Gradient Methods
Let y denote the response space, then ne(- x)) A()) admits a probability distribution over V. and the policy gradient is given by
V VeJpolccy (0) Eg-D D Σ [[[ ((()))) log Tzet (3) x)) B VoToly
To (
x))
yEy
Since To x:) is a probability, it must have Eyey Vono(y x)) 0, so we can add any baseline
term 6(z) independent of y to reduce the variance without affecting the exact policy gradient:
V VoJpolicy (0) E
(- B.wD..yvaotte
(R(æ, y) b(x)) Vo log moly x))]
(12)
advantage A(
where R(x, y) == r(x, y) - 3 log = - denotes the regularized reward. The policy gradient
12
can be estimated by rolling out x ~ D and y T- x), estimating the advantage A(æ y) R(x, y) b(x)) and taking the average. The empirical policy loss at each step is then written
Epolicy (0)
1 B
A(z,, v.) log Toly; xx)).
(13)
To reduce the variance, the baseline usually set as the value function (expected reward under current policy), which is approximated by a critic model as in PPO (39, or by Monte-Carlo (MC) samples as in RL.OO I and GRPO 43 this paper, we focus on the latter approach, which has more transparent form.
23

---

C Experiment Details
In this section, we provide additional experimental details about SNGP-PM training, RLOO training and policy evaluations.
C.1 SNGP-PM Training
Our implementation of SNGP follows 37 particularly the one applied to Bert Specifically, we only apply spectral normalization to the linear layer in the last decoder and set the spectral normalization range to 1. We set the random feature size D, = 4096, identical to the hidden size D, 4096 of the Llama-3.1-881 Instructttt model. We set the amplitude σą = 1, scaling factor = 10, and ridge coefficient т 0.001. We update all weights during training, except the random feature weights and in the GP layer. For the baseline PM, we directly apply a linear layer on the last hidden states of dimension Dj, and update all weights during training. For training, follow the code from OpenRLHF which is an easy-to-use use. high-pert performance open-source RLHF framework [20]. Hyperparameters summarized in Table 7 are evaluated on HelpSteer2-Preffe Preffrence validation set [51] Reward Bench [25] and RMM-Bench [31. and the results shown in Table 8 We present the results with learning rate 4e-6 in Table T aas they achieve the highest accuracy on the validation set.
PM and SNGP-PM trained with different learning rates
Table 7: Training configurations for PM and SNGP-PM.
Item
Value
Base model name
Batch size
256
Micro batch size
16
Training epochs
2 (3 if counting the covariance calculation pass)
Quantization
BFloat16
Learning rate (LR)
[2e-6, 3e-6, 4e-6, 5e-6, 6c-6)
Learning rate scheduler
Cosine with min LR (0.1 x base LR) 0.03
Warm up ratio
Gradient accumulation steps
16
Max input length
8192
DeepSpeed Zero stage
2
Flash attention
Enabled
Table 8: Performance of PM and SNGP-PM trained with varying learning rates.
Validation (%)
Reward Bench (%)
RM Beneh (%)
Model
avg.
hard
safety
reasoning
avg.
code
safety
30-6
80.1
87.8
96.6
73.6
61.0
53.8
91.8
4e-6
80.1
87.7
96.4
73,1
89.4
91.8
68.7
67.0
60.5
55.1
92.3
PM
5e-6
78.7
87.9
97.8
72.1
89.1
92.5
68.4
65.3
61.5
54.8
92.1
6c-6
77.3
87.9
97.2
72.1
67.7
54.9
26-6
77.8
86.8
96
72.5
52.6
SNGP-PM
36-6
78.1
87.1
95.8
71.2
89.2
92.0
68.1
65.7
60.1
53,3
93.1
4e-6
79.3
87.3
95.8
73.8
89.4
90.0
68.0
67.1
59.5
54,2
91.2
5c-6
76.7
86.3
96.6
70.1
67.4
53.9
C.2 RLOO Training
For the Ultrafeedback prompt dataset, extract the prompts from the preference version used in 21 Given resource constraints, we sample the first 33% of the dataset, which consists of 19,456 prompts covering wide range of domains. Our implementation of RLOO follows OpenRLHF
intent/sngp.pp
ul brrafeebbackkoveraall.

---

20), with modifications serving the SNGP-PM with router instead of the standard pointwise RM. In particular, we move the advantage computation from the trainer the PM side. We run experiments on NNNIII A A100-80G GPUs, using 2 GPUs serving the PMs in parallel, co-locating the actor and reference models on 4 GPUs, and using the remaining GPUs for on-policy sampling. For LLM-as----udge we call a remotely hosted DeepSeek-R1 which allows 200 requests per minute. We use the prompt template in Appendix A and convert the returned label to a reward difference in {- 0, 2). Training hyperparameters are summarized in Table 9
Table 9: Training configurations for RLOO.
Item
Value
Base model name
Llama-3.1 -.B--stnnttttt
Rollout batch size
1024
Train batch size
1024
Micro rollout batch size
128
Micro train batch size
8
Training episodes
Max prompt length
2048
Max generation length
1024
Quantization
BF
Actor learning rate
5e-7
KL coefficient
0.01
Advantage clip ratio
0.2
Samples per prompt (K in RLOO)
4
Speed Zero stage
Flash attention
Enabled
C.3 Policy Model Evaluations
For Arena-Hard-v0.1 (26), we use the official library adopting the default decoding configu- ration and comparing the WR against GPT-4-0314, using GPT-4.1 as the judge. For AlpacaFE- val 2.0 12]. follow the default setting evaluating the WR against GPT-4-Turbo using weighted alpaca_eval _arla__vll____e___l_aa__eee____e _trrboourooorrr as annotator. When generating the output, we use the default generation configuration of average WR and length-controlled (LC) WR. For MT-Bench [62, we follow an open codebase 19) and update the chat format for compatibility with the Llama-3.1 1-8.a...------ttuuu chat template. We use GPT-4-Turhe Turbo as the judge to rate the quality of responses with scalar scores ranging from 1 to 10. Detailed results on Arena-Haan Arena-Hard-vOl Hard-v0.1 and AlpacaEval 2.0 are provided in Tables [0land]] II
We run the evaluation times and report the
Uncertainty
88.1% (DeepSeek-R)
Uncertainty
Random
Random
RM Berch Accuracy
29.0
28.5
20:1% (No Routing)
Arano Hard
27.5
27.0
668 07 (No Routing)
100.0
Percentage of Calls (96)
Percentage of Calls (56)
Figure 3: Uncertainty-base routing outperforms random routing with the same number of strong
LLM judge calls on preference accuracy (R.M-Bench) and downstream alignment (Arena-Hard-v0.1)
httt:p////gggtaaooorreemmrmmamaa auto
config.
https:// https://githuu:
MT- Bench
25

---

Table 10: Performance comparison of downstream policy models on Arena- Arena-Hard-van Hard-v0.1 with GPT-4.1 as judge. WR stands for win rate against GPT-4-0314, and CI stands for confidence interval.
Arena-Hard-vO.J (%)
Model
Num of Calls
WR
CI
Base model
24.5
(-1.8/+1.)
No routing
28.1
(-1.9/+16)
Uncertainty (1.35)
7668 (6.56%)
28.9 (+2.4)
(.2.4/+17)
Uncertainty (1.30)
10522 (9.01%)
28,9 (+1.2)
(-2.0/+2.0)
Uncertainty (1.20)
21363 (18.3%)
29.8 (+1.3)
(-2.2/+19)
Random (1.35)
7523 (6.40%)
26.5
(-1.5/+1.5)
Random (1.30)
10854 (9. 99.3%%
27.7
(-1.8/+19)
Random (1.20)
20474 (17.5%)
28.5
(-2.0/+18))
Table 11: Performance comparison of downstream policy models on AlpacaEval 2.0 with GPT-4 Turbo as judge. LCWR is the length-controlled win rate, and WR is the win rate. Avg Length shows average generation length.
AlpacaEval 2.0
Model
Num of Calls
LCWR (%)
WR (%)
Avg. Length
Base model
22.31
23.63
2304
No routing
25.40
27.35
2142
Uncertainty (1.35)
7668 (6.56%)
26.28 (+0.58)
28.97 (+0.42)
2133
Uncertainty (1.30)
10522 (9.01%) %)
26.34 (+1.20)
28.53 (+0.24)
2163
Uncertainty (1.20)
21363 (18.3%)
26.45 (+0.47)
28.91 (+0.40)
2167
Random (1.35)
7523 (6.40%)
25.70
28.55
2085
Random (1.30)
10854 (9.30%)
25.14
28.29
2157
Random (1.20)
20474 (17.5%)
25.98
28.51
2189
C.4 Judge Latency
The judge latency depends model and its serving efficiency, well other engineering factors in the pipeline. our experiment, we use DeepSeek-RR R1 through API calls, which hosted and allows 200 requests per minute (RPM). In RLHF experiment, the batch size is 256, and generate samples per prompt, resulting in 1536 total pairwise comparisons per batch, and only about 6% to 9% 18% of them routed to the judge (see Table corresponding to 92 to 138 to 276 judge requests per batch. Therefore, most the requests within a batch can be executed complete parallel (no backlog), and the overhead is further reduced given that requests are sent asynchronously. Notably, our use of 200 RPM-limited DeepSeek-RI is due to the early time when experiments are conducted and the budget constraints. In current practice, various strong LLMs be used, possibly with higher rate limit. In practical online RLHF setting, the routing framework can be further optimized by dynamically adjusting the uncertainty threshold to ensure the number of judge requests stays within the limit, preventing the pipeline from blocking.
26


Reviews:
[{"review_id": "0V5pULn7JT", "reviewer": "Reviewer_WJxQ", "summary": "This paper introduces an uncertainty-based routing framework for improving reward model (RM) performance in RLHF settings. The central idea is to quantify the RM’s uncertainty using a pairwise preference model (PM) augmented with Spectral-normalized Gaussian Process (SNGP) uncertainty estimation. When the RM is uncertain about a pairwise comparison, the sample is routed to the more reliable but slower LLM judge; otherwise, the RM’s output is used directly. This system is evaluated on standard reward modeling benchmarks and downstream alignment tasks, showing that routing based on uncertainty leads to higher accuracy and improved policy alignment compared to random routing under similar resource constraints.", "questions": "See weaknesses plz.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: The topic is aligned with the conference community. This paper has solid technical motivations and effective methods for routing. It has comprehensive experimental evaluation, which makes the conclusion persuasive.\n This paper briefly acknowledges the limitations and proposes reasonable future improvements.\n\nWeaknesses:\n\n•\tInadequate Analysis of Judge Latency in Large-Scale RLHF: While cost and parallelism are mentioned (Tables 3 and 5), there isn't a detailed operational account of potential delays in practical large-scale RLHF settings where many judge calls may backlog.\n\n•\tLimited Diversity in LLM Judge Usage: The limited diversity in LLM judge usage is a notable aspect of this work. All experiments rely on DeepSeek-R1, which, as the authors acknowledge, is not specifically optimized for judging and requires significant resources. I wonder if the authors explored the potential of utilizing other LLMs, particularly lightweight models, as judges. While the absolute performance might not match DeepSeek-R1, exploring the trade-off between effectiveness and efficiency could yield valuable insights and would be a worthwhile addition to the experimental analysis.", "was_revised": true, "final_justification": "Considering the comments of other reviewers, I maintain my positive rating. But I would like to increase the Clarity score by one point. I hope that the revised version will include the discussion in Section 5 and the Appendix, as the authors have promised.", "rebuttal": "We appreciate the reviewer for their positive assessment and insightful feedback. We address the Weaknesses below.\n\n> W1:  Inadequate Analysis of Judge Latency in Large-Scale RLHF: While cost and parallelism are mentioned (Tables 3 and 5), there isn't a detailed operational account of potential delays in practical large-scale RLHF settings where many judge calls may backlog.\n\nThank you for this question. While our experiments in Tables 3 and 5 aimed to provide a comparative analysis of the time cost between uncertainty and random routing strategies, we agree that a more detailed operational account of managing judge latency is a valuable addition. \n\nIn particular, the latency depends on which judge is used and how it is served, as well as other engineering factors in the pipeline. In our experiment, we use DeepSeek-R1 through api calls, which is hosted by a cloud provider that allows 200 requests per minute (RPM), and it can take from 20 seconds to above 1 minute to generate an output depending on the reasoning length. In our RLHF experiment, the batch size is 256, and we generate 4 samples per prompt, resulting in 1536 total pairwise comparisons per batch, and only about 6% to 9% to 18% of them are routed to the judge (see Table 6), corresponding to 92 to 138 to 276 judge requests per batch. Therefore, most of the requests within a batch can be executed in complete parallel (no backlog), and the overhead is further reduced given that requests are sent and completed asynchronously. \n\nNotably, our use of 200 RPM limited DeepSeek-R1 is due to the early time when experiments are conducted and the budget constraints. In current practice, there are various strong LLMs that can be used, possibly with a higher rate limit. For example, tier 1 user can have 500 RPM for GPT o3/o3-pro and 1000 RPM for o3-mini. \n\nIn a practical online RLHF setting, the routing framework can be further optimized by dynamically adjusting the uncertainty threshold $u$ to ensure the number of judge requests stays within the limit, preventing the pipeline from blocking.\nWe will add a discussion of these operational considerations to the appendix to provide a clearer picture for practitioners looking to implement this framework.\n\n> W2: Limited Diversity in LLM Judge Usage: The limited diversity in LLM judge usage is a notable aspect of this work. All experiments rely on DeepSeek-R1, which, as the authors acknowledge, is not specifically optimized for judging and requires significant resources. I wonder if the authors explored the potential of utilizing other LLMs, particularly lightweight models, as judges. While the absolute performance might not match DeepSeek-R1, exploring the trade-off between effectiveness and efficiency could yield valuable insights and would be a worthwhile addition to the experimental analysis.\n\nWe thank the reviewer for raising this point. We would like to clarify that our uncertainty-based routing strategy is generic and can be applied to any LLM judge. In particular, our judge prompt in Appendix B is not specifically tailored for any specific LLM. Our choice of DeepSeek-R1 is due to its high capability within our budget constraint. We believe that stronger models and specialized judges can be used to improve the reward signals.\n\nNevertheless, the primary goal of our work is to establish a strong proof-of-concept for the uncertainty-routing framework. By using the powerful, high-accuracy R1 judge, we measure the potential of improvement one could gain from routing to a strong oracle. If the framework showed no significant benefit with a powerful judge, its utility would be questionable.\n\nHaving established this viability, exploring more resource-efficient judges is a logical next step for future work. As briefly noted in our conclusion, one could replace it with a smaller-scale generative RM to further improve the judge quality and inference efficiency. This opens up the possibility of a hierarchical routing system, where a local fast RM handles the most certain pairs, a lightweight, efficient judge (e.g., a fine-tuned 2B generative RM) handles cases of moderate uncertainty, and an expensive but powerful judge is reserved for only the most uncertain and critical cases. This approach would offer a more nuanced balance between cost, latency, and feedback quality. We will expand this discussion in Section 5 to highlight the potential of diverse LLM judge usage."}, {"review_id": "iLZMMuqCS6", "reviewer": "Reviewer_Qbyt", "summary": "This paper proposes an uncertainty based scheme that queries a strong LLM to label preference in RL/PG training of LLM. The algorithm first trains a pairwise RM and estimates its uncertainty using GP. At policy training time, if the uncertainty on a given pair of trajectories (of a batch sampled by the policy) is higher than a threshold, then the strong model is queried to provide a label; otherwise, the learned RM is used. The authors show the proposed improves the reward prediction quality and is more efficient than random routing. In addition, they show policy learning from this LLM-assisted reward improves the policy performance.", "questions": "1. In Eq (8) why is p(x,y_1,y_2) defined as g/u instead of g?\n\n2. How are the hyperparameters of SNGP part chosen?\n\n3. How much more tokens does the Uncertainty-based method use than Random routing in the LLM training experiments? In the previous RM experiments, it suggest Uncertainty-based generates more tokens.\n\n4. The current highlights in Table 6 is a bit misleading. e.g. in MT-Bench Random (1.35) performs on-part with Uncertainty (1.3) on Avg and better than the rest except No routing on Turn 1. I would suggest highlighting just one row per column. \n\n5. Can you explain why the improvement of proposed methods vary a lot from different benchmarks in Table 6? E.g. how does it relate to the ability of LLM to judge? In particular, in Arena-Hard and MT-bench, the results are not much better than the baseline of no-routing, especially given the noises, I'm not sure if the current results are statistically significant.\n\n6. Please add a discussion on literature of active learning, especially in the context of RL.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "S1. The paper is well motivated and RLAIF or RL with LLM is an important area now. \n\nS2. The proposed algorithm seems straightforward and scalable. The current experimental results seem promising.\n\nS3. The paper is overall well written. \n\nS4. The paper reveals a good insight on the difference between pointwise and pairwise RM in uncertainty estimation,\n\nW1. The paper lacks discussion of the active learning literature, which has a large corpus of similar works, and some are for RL, though not necessarily for LLM training. \n\nW2 The improvement is LLM training is not super significant.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the positive feedback and for acknowledging the strengths of our work, and we would like to answer the questions in the following. \n\n> Q1: In Eq (8) why is $p(x,y_1,y_2)$ defined as $g/u$ instead of $g$?\n\nEq (8) comes from the mean-field approximation to the Gaussian posterior, as presented in Eq (19) in the original SNGP paper [1]. Intuitively, the preference score $p$ encompasses both aleatoric and epistemic uncertainties, while $g$ only conveys in-distribution aleatoric uncertainty through its scale. Thus, by dividing $u$ conveying epistemic uncertainty, the resulting score $p$ is less overconfident on OOD samples. \n\n> Q2: How are the hyperparameters of SNGP part chosen?\n\nThe SNGP hyperparameters are chosen based on the recommended tuning strategy in [1] (Appendix A.2). In particular, we choose the spectral norm bound ensuring that the validation accuracy remains close to vanilla PM. The GP dimension is chosen to match the model's (Llama-3.1-8B-Instruct) hidden dimension. Kernel amplitude and scaling factors do not affect accuracy but only impact the range of scores and uncertainties; thus, we tune them so that the scores' range matches vanilla PM and choose the uncertainty router threshold accordingly. Ridge coefficient is insignificant in our experiments, so we just set it as recommended in [1]. \n\n> Q3: How much more tokens does the Uncertainty-based method use than Random routing in the LLM training experiments? \n\nWe thank the reviewer for this good question. While uncertainty-based routing consumes more tokens for the same number of routed pairs, it actually uses *less* tokens to achieve the same prediction accuracy. In particular, by combining tables 2 and 3 (also 4 and 5), we can plot accuracy vs inference time curves and observe that uncertainty-based routing uses approximately 55% less time than random routing to achieve 89.1% accuracy on RewardBench, and approximately 16% less time to achieve 77.1% accuracy on RMBench in our setting. Thus, given that the number of tokens scales with inference time, our experiments suggest that fewer tokens are required by uncertainty-based routing. \n\n> Q4: The current highlights in Table 6 is a bit misleading. I would suggest highlighting just one row per column.\n\nThank you for the suggestion, and we will revise it accordingly, with one highlighting row per column.\n\n> Q5: Can you explain why the improvement of proposed methods vary a lot from different benchmarks in Table 6? E.g. how does it relate to the ability of LLM to judge? In particular, in Arena-Hard and MT-bench, the results are not much better than the baseline of no-routing, especially given the noises, I'm not sure if the current results are statistically significant.\n\nThank you for the question. Firstly, we would like to highlight that the benchmarks in Table 6 have different focuses. \n\n- Arena-Hard focuses on challenging user prompts (open-ended software engineering problems, math questions, etc) where the win rate (WR) is the metric. \n- AlpacaEval focuses on general helpfulness and human instruction following, where the length-controlled win rate (LCWR) is more important, as the naive WR may possess length bias and has less correlation with real-world human preference.\n- MT-Bench focuses on testing the multi-turn dialog ability, where the Turn 2 score (also average score) is more important. \n\nGiven the nature of these benchmarks, we aim to compare our uncertainty-routing method with no/random routing mainly on the most important metrics, namely Arena-Hard WR, AlpacaEval LCWR, and Turn 2 score in MT Bench. On all of these 3 metrics, we observe consistent improvement over the baselines, thus showing the efficacy of our method. \n\nRegarding the performance variation, we attribute it to the following factors:\n\n1) Benchmark and judge mismatch: Benchmark metrics reflect different, sometimes even conflicting skills, and the evaluation judges (like GPT-4.1 and GPT-4-Turbo) differ from the DeepSeek-R1 judge used during training, creating variance.\n2) Multi-objective trade-offs: On MT-Bench, our method slightly lowers the Turn 1 score but improves the more difficult Turn 2 score, leading to a better conversational model.\n\n\n> Q6: The paper lacks discussion of the active learning literature.\n\nWe thank the reviewer for this valuable suggestion. We agree that our work has strong connections to active learning and that a more detailed discussion would better contextualize our contributions. In Section 5, we briefly allude to this by suggesting that uncertain samples could be sent to human annotators to augment the training data.\nTo make this connection explicit, we will expand this discussion in the revision with the following paragraph:\n\n\"Our uncertainty-based routing framework is conceptually aligned with active learning, which seeks to improve data efficiency by strategically selecting the most informative instances for labeling by an oracle [4]. In our work, the strong LLM judge acts as the oracle, and our uncertainty score serves as the acquisition function to identify the most uncertain pairs of responses. Classic query strategies in active learning often rely on model uncertainty, such as selecting the least confident predictions or using a committee of models to find contentious examples [5], which have been extended to deep learning through Bayesian methods [6]. While active learning has been explored for learning reward/preference functions from limited interaction in the RL context [7,8], our primary contribution is the application of this principle to a direct online RLHF setting. Here, the feedback from the oracle is used immediately to construct advantage estimates for policy gradient updates, and feeding these newly labeled high-uncertainty pairs back into the reward model for continued training is a natural extension that would constitute a full active learning cycle.\"\n\n\n[1] Liu, Jeremiah Zhe, et al. \"A simple approach to improve single-model deep uncertainty via distance-awareness.\" JMLR 2023.\n\n[2] Li, Tianle, et al. \"From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline.\" arXiv preprint arXiv:2406.11939 (2024).\n\n[3] Dubois, Yann, et al. \"Length-controlled alpacaeval: A simple way to debias automatic evaluators.\" arXiv preprint arXiv:2404.04475 (2024).\n\n[4] Settles, Burr. \"Active learning literature survey.\" (2009).\n\n[5] Seung, H. Sebastian, Manfred Opper, and Haim Sompolinsky. \"Query by committee.\" COLT 1992.\n\n[6] Gal, Yarin, Riashat Islam, and Zoubin Ghahramani. \"Deep bayesian active learning with image data.\" ICML 2017.\n\n[7] Houlsby, Neil, et al. \"Bayesian active learning for classification and preference learning.\" arXiv preprint arXiv:1112.5745 (2011)..\n\n[8] Daniel, Christian, et al. \"Active Reward Learning.\" Robotics: Science and systems. Vol. 98. 2014."}, {"review_id": "l5Nim5nPMu", "reviewer": "Reviewer_Hyxs", "summary": "This paper is on reward models (RMs) in the context of reinforcement learning for fine-tuning large language models. The main contribution is an uncertainty-based routing framework designed to improve the efficiency and quality of the fine-tuning process. The proposed method evaluates the uncertainty of its internal reward model. When the RM's confidence is low on a given sample, the framework defers the decision to a more powerful, albeit costly, external LLM judge. This dynamic allocation aims to leverage the high-quality signal from the stronger judge only when necessary, thus optimizing the trade-off between cost and performance. The authors validate their approach by applying the routing strategy to fine-tune a Llama 8B model, demonstrating its effectiveness empirically.", "questions": "The paper's main contribution is an uncertainty-based routing strategy, which I find promising. \nMy primary questions revolve around including more UQ methods, as this could further strengthen the paper's claims.\nThe framework's effectiveness is fundamentally tied to the reliability of its uncertainty estimates. \n\n- Have the authors considered or experimented with other well-established and computationally inexpensive UQ techniques for the reward model? A broader comparison could provide a more robust foundation for the paper.\n\nPossible UQ techniques include:\n- MC Dropout: a relatively simple yet powerful technique to implement, especially for the classification head of the reward model. It would be an interesting and straightforward comparison.\n- Laplace Approximation: A last-layer Laplace approximation is another strong candidate for this use case. Since the final classification head likely has a manageable number of parameters (d×1), a full Laplace approximation on this layer might even be feasible and could provide a more principled uncertainty estimate. Could the authors comment on the potential of using such a method within their framework?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The core concept of the paper is well-motivated and intuitive. Using model uncertainty to dynamically route queries between a local, efficient RM and a more capable external judge is a logical approach to balancing signal quality and computational cost.\n- The work provides a valuable direction for future research. While not the first to use uncertainty quantification, its application as an explicit routing mechanism in a hybrid system is a practical and interesting contribution that could inspire further work in efficient LLM alignment.\n- The empirical results effectively demonstrate the value of the proposed method. The comparison against a random routing baseline clearly shows that the uncertainty-based strategy is superior, making a case for its adoption.\n\nWeaknesses:\n- The exploration of different uncertainty quantification (UQ) techniques is limited to a single method. Given that the performance of the entire framework hinges on the quality of the uncertainty estimates, a more thorough investigation of alternative UQ methods would be beneficial. This is also because there is a vast literature on UQ techniques for neural networks that can be used. \n- The scope of the experimental validation is somewhat limited. The experiments are conducted only on a Llama 8B model; while this is a significant undertaking, demonstrating the framework's effectiveness on even larger, state-of-the-art models would make the conclusions more generalizable.", "was_revised": true, "final_justification": "I thank the author for the rebuttal. I maintain the positive score.", "rebuttal": "We appreciate the reviewer for the positive feedback and highlighting the promising nature of our uncertainty-based routing strategy. We address their concerns in the following.  \n\n> W1&Q1: The exploration of different uncertainty quantification (UQ) techniques is limited to a single method. Given that the performance of the entire framework hinges on the quality of the uncertainty estimates, a more thorough investigation of alternative UQ methods would be beneficial. This is also because there is a vast literature on UQ techniques for neural networks that can be used.\n> Have the authors considered or experimented with other well-established and computationally inexpensive UQ techniques for the reward model? A broader comparison could provide a more robust foundation for the paper?\n\nThank you for raising this concern/question, and we agree that a broader comparison of UQ techniques is an important research direction. That said, we would like to highlight that our main contribution is introducing the uncertainty routing framework for the reward model, enabling more reliable reward signals and better online RLHF in a cost-efficient way. Within this framework, various UQ methods can be applied, and we believe a good UQ method can potentially lead to better performance. \n\nWith budget constraint and limited capacity (we conduct experiments not only on reward benchmarks but also downstream alignment), we are not able to explore many other UQ methods. We chose SNGP for proof-of-concept because of its: (1) low inference cost; (2) implementation simplicity; (3) principled design. Firstly, SNGP only requires a single forward pass at inference time for both prediction and uncertainty score, which is a desired property as an RM must evaluate many response pairs quickly in the online RLHF pipeline. Secondly, its implementation is relatively simple, and it does not require much modification to the standard reward learning pipeline. Thirdly, it is designed to be distance-aware, making it particularly well-suited for detecting OOD samples that are the central challenge for reliable reward.\n\nRegarding the UQ methods mentioned in the question: \n\n- MC Dropout: We agree that MC Dropout is a straightforward and powerful technique, and we tried it in the early stage to examine the relation between accuracy and uncertainty. However, it requires multiple forward passes at inference time to estimate uncertainty. In our setting, this would multiply the inference cost substantially, even without counting the judge calls, creating a latency bottleneck that conflicts with the efficiency requirements of an online RLHF pipeline. Therefore, we view it as less suitable for our framework compared to a single-pass method like SNGP.\n- Laplace Approximation: Last-layer Laplace approximation is closely related to our method, as SNGP also implements Laplace approximation. As detailed in the original SNGP paper [1], the method replaces the final dense layer with an approximate Gaussian Process and computes the posterior using the Laplace approximation on its random feature expansion. This provides a principled Bayesian perspective that yields a closed-form posterior, which can be learned scalably with minimal modification to the standard training pipeline. While a direct Laplace approximation on the final layer is another valid approach, our chosen method, SNGP, allows us to scale similarly efficiently while leveraging strong theoretical properties, serving as a good baseline.\n\n> W2: The scope of the experimental validation is somewhat limited. The experiments are conducted only on a Llama 8B model; while this is a significant undertaking, demonstrating the framework's effectiveness on even larger, state-of-the-art models would make the conclusions more generalizable.\n\nFirstly, we would like to highlight that our experiment is not limited to testing reward accuracy on benchmarks (Reward Bench and RM-Bench) but also includes the improvement brought to downstream policy model alignment, which makes our validation rather comprehensive. \n\nWe agree that demonstrating the framework's effectiveness on larger models is an important step for generalization. Our experiments, which involve computationally intensive online RLHF, were designed to provide a robust proof-of-concept for the uncertainty-routing framework. We chose the Llama-3.1-8B model as a representative foundation to establish this initial validation within the budget constraint. We consider scaling these experiments to larger models for future work.\n\n[1] Liu, Jeremiah Zhe, et al. \"A simple approach to improve single-model deep uncertainty via distance-awareness.\" JMLR 2023."}]

Author rebuttal:

Meta-review: This paper proposes that a lightweight reward model (in RLHF) be equipped with an uncertaintly quantification method that decides when the RM is uncertain about a particular generation and therefore sends it to a larger/stronger model for judgement. The method is shown to improve over equivalent baselines in downstream policy effectiveness.

Reviewers agreed that the approach is an elegant and principled way to solve the problem of doing RLHF while optimizing resources. The topic is a very important one as it addresses the RLHF pipeline that is ubiqutious in industry. The experimental results show positive gains, but reviewers did point out a few different dimensions that ideally would have been explored more (e.g. different UQ methods, different Judge LM's etc), but these experiments would be expensive so maybe out of scope. A larger concern I have personally is that measuring success through # of calls to the RM may not be a perfect measure of efficiency since an RM equipped with a UQ method (like SNGP) will be somewhat more FLOP-intensive. iso-flops measurements may have been more helpful.

Still, all reviewers found the contribution good and it is worth acceptance.
Decision: Accept (poster)