Title: Ask a Strong LLM Judge when Your Reward Model is Uncertain
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

RLHF

Reward Models

LLM-as-a-Judge
Abstract: Reward model (RM) plays a pivotal role in reinforcement learning with human feedback (RLHF) for aligning large language models (LLMs). However, classical RMs trained on human preferences are vulnerable to reward hacking and generalize poorly to out-of-distribution (OOD) inputs. 
By contrast, strong LLM judges equipped with reasoning capabilities demonstrate superior generalization, even without additional training, but incur significantly higher inference costs, limiting their applicability in online RLHF. 
In this work, we propose an uncertainty-based routing framework that efficiently complements a fast RM with a strong but costly LLM judge. Our approach formulates advantage estimation in policy gradient (PG) methods as pairwise preference classification, enabling principled uncertainty quantification to guide routing. Uncertain pairs are forwarded to the LLM judge, while confident ones are evaluated by the RM. Experiments on RM benchmarks demonstrate that our uncertainty-based routing strategy significantly outperforms random judge calling at the same cost, and downstream alignment results showcase its effectiveness in improving online RLHF.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is on reward models (RMs) in the context of reinforcement learning for fine-tuning large language models. The main contribution is an uncertainty-based routing framework designed to improve the efficiency and quality of the fine-tuning process. The proposed method evaluates the uncertainty of its internal reward model. When the RM's confidence is low on a given sample, the framework defers the decision to a more powerful, albeit costly, external LLM judge. This dynamic allocation aims to leverage the high-quality signal from the stronger judge only when necessary, thus optimizing the trade-off between cost and performance. The authors validate their approach by applying the routing strategy to fine-tune a Llama 8B model, demonstrating its effectiveness empirically.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The core concept of the paper is well-motivated and intuitive. Using model uncertainty to dynamically route queries between a local, efficient RM and a more capable external judge is a logical approach to balancing signal quality and computational cost.\n- The work provides a valuable direction for future research. While not the first to use uncertainty quantification, its application as an explicit routing mechanism in a hybrid system is a practical and interesting contribution that could inspire further work in efficient LLM alignment.\n- The empirical results effectively demonstrate the value of the proposed method. The comparison against a random routing baseline clearly shows that the uncertainty-based strategy is superior, making a case for its adoption.\n\nWeaknesses:\n- The exploration of different uncertainty quantification (UQ) techniques is limited to a single method. Given that the performance of the entire framework hinges on the quality of the uncertainty estimates, a more thorough investigation of alternative UQ methods would be beneficial. This is also because there is a vast literature on UQ techniques for neural networks that can be used. \n- The scope of the experimental validation is somewhat limited. The experiments are conducted only on a Llama 8B model; while this is a significant undertaking, demonstrating the framework's effectiveness on even larger, state-of-the-art models would make the conclusions more generalizable.\n\nQUESTIONS:\nThe paper's main contribution is an uncertainty-based routing strategy, which I find promising. \nMy primary questions revolve around including more UQ methods, as this could further strengthen the paper's claims.\nThe framework's effectiveness is fundamentally tied to the reliability of its uncertainty estimates. \n\n- Have the authors considered or experimented with other well-established and computationally inexpensive UQ techniques for the reward model? A broader comparison could provide a more robust foundation for the paper.\n\nPossible UQ techniques include:\n- MC Dropout: a relatively simple yet powerful technique to implement, especially for the classification head of the reward model. It would be an interesting and straightforward comparison.\n- Laplace Approximation: A last-layer Laplace approximation is another strong candidate for this use case. Since the final classification head likely has a manageable number of parameters (d×1), a full Laplace approximation on this layer might even be feasible and could provide a more principled uncertainty estimate. Could the authors comment on the potential of using such a method within their framework?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an uncertainty based scheme that queries a strong LLM to label preference in RL/PG training of LLM. The algorithm first trains a pairwise RM and estimates its uncertainty using GP. At policy training time, if the uncertainty on a given pair of trajectories (of a batch sampled by the policy) is higher than a threshold, then the strong model is queried to provide a label; otherwise, the learned RM is used. The authors show the proposed improves the reward prediction quality and is more efficient than random routing. In addition, they show policy learning from this LLM-assisted reward improves the policy performance.\n\nSTRENGTHS AND WEAKNESSES:\nS1. The paper is well motivated and RLAIF or RL with LLM is an important area now. \n\nS2. The proposed algorithm seems straightforward and scalable. The current experimental results seem promising.\n\nS3. The paper is overall well written. \n\nS4. The paper reveals a good insight on the difference between pointwise and pairwise RM in uncertainty estimation,\n\nW1. The paper lacks discussion of the active learning literature, which has a large corpus of similar works, and some are for RL, though not necessarily for LLM training. \n\nW2 The improvement is LLM training is not super significant.\n\nQUESTIONS:\n1. In Eq (8) why is p(x,y_1,y_2) defined as g/u instead of g?\n\n2. How are the hyperparameters of SNGP part chosen?\n\n3. How much more tokens does the Uncertainty-based method use than Random routing in the LLM training experiments? In the previous RM experiments, it suggest Uncertainty-based generates more tokens.\n\n4. The current highlights in Table 6 is a bit misleading. e.g. in MT-Bench Random (1.35) performs on-part with Uncertainty (1.3) on Avg and better than the rest except No routing on Turn 1. I would suggest highlighting just one row per column. \n\n5. Can you explain why the improvement of proposed methods vary a lot from different benchmarks in Table 6? E.g. how does it relate to the ability of LLM to judge? In particular, in Arena-Hard and MT-bench, the results are not much better than the baseline of no-routing, especially given the noises, I'm not sure if the current results are statistically significant.\n\n6. Please add a discussion on literature of active learning, especially in the context of RL.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an uncertainty-based routing framework for improving reward model (RM) performance in RLHF settings. The central idea is to quantify the RM’s uncertainty using a pairwise preference model (PM) augmented with Spectral-normalized Gaussian Process (SNGP) uncertainty estimation. When the RM is uncertain about a pairwise comparison, the sample is routed to the more reliable but slower LLM judge; otherwise, the RM’s output is used directly. This system is evaluated on standard reward modeling benchmarks and downstream alignment tasks, showing that routing based on uncertainty leads to higher accuracy and improved policy alignment compared to random routing under similar resource constraints.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The topic is aligned with the conference community. This paper has solid technical motivations and effective methods for routing. It has comprehensive experimental evaluation, which makes the conclusion persuasive.\n This paper briefly acknowledges the limitations and proposes reasonable future improvements.\n\nWeaknesses:\n\n•\tInadequate Analysis of Judge Latency in Large-Scale RLHF: While cost and parallelism are mentioned (Tables 3 and 5), there isn't a detailed operational account of potential delays in practical large-scale RLHF settings where many judge calls may backlog.\n\n•\tLimited Diversity in LLM Judge Usage: The limited diversity in LLM judge usage is a notable aspect of this work. All experiments rely on DeepSeek-R1, which, as the authors acknowledge, is not specifically optimized for judging and requires significant resources. I wonder if the authors explored the potential of utilizing other LLMs, particularly lightweight models, as judges. While the absolute performance might not match DeepSeek-R1, exploring the trade-off between effectiveness and efficiency could yield valuable insights and would be a worthwhile addition to the experimental analysis.\n\nQUESTIONS:\nSee weaknesses plz.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}