Title: Rectifying Shortcut Behaviors in Preference-based Reward Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning with Human Feedback

Shortcut Learning

Reward Hacking
Abstract: In reinforcement learning from human feedback, preference-based reward models play a central role in aligning large language models to human-aligned behavior. However, recent studies show that these models are prone to reward hacking and often fail to generalize well due to over-optimization. They achieve high reward scores by exploiting shortcuts, that is, exploiting spurious features (e.g., response verbosity, agreeable tone, or sycophancy) that correlate with human preference labels in the training data rather than genuinely reflecting the intended objectives. In this paper, instead of probing these issues one at a time, we take a broader view of the reward hacking problem as shortcut behaviors and introduce a principled yet flexible approach to mitigate shortcut behaviors in preference-based reward learning. Inspired by the invariant theory in the kernel perspective, we propose Preference-based Reward Invariance for Shortcut Mitigation (PRISM), which learns group-invariant kernels with feature maps in a closed-form learning objective. Experimental results in several benchmarks show that our method consistently improves the accuracy of the reward model on diverse out-of-distribution tasks and reduces the dependency on shortcuts in downstream policy models, establishing a robust framework for preference-based alignment.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a unified shortcut-mitigation framework named PRISM, Preference-based Reward Invariance for Shortcut Mitigation, to improve the robustness and generalization of reward models in reinforcement learning from human feedback (RLHF). By refactoring reward hacking problems as shortcut behaviors, the authors model spurious correlations (i.e., verbosity, tone, sycophancy) as group actions. Key contribution is to learn various shortcut features as group-invariant kernels, efficiently approximated using random feature maps. Theoretical generalization bounds and empirical results show improvements on out-of-distribution (o.o.d) preference datasets and downstream alignment tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- This paper is well-written and easy to follow. The idea of reframing reward hacking as a form of shortcut learning is timely and highly relevant to current challenges in LLM alignment.\n- Unlike prior works that focus on individual shortcuts in isolation, PRISM is general and supports multi-bias mitigation under a unified objective.\n- The use of group-invariant kernels and the approximation via feature maps is mathematically well-motivated.\n\nWeaknesses\n- The empirical evaluation is somewhat limited in scope. While PRISM is compared against standard baselines, comparisons to recent and closely-related methods such as reward ensembles, multi-objective reward models, and robust reward modeling approaches (i.e., [1], [2], [3])  are missing. Some of these methods (i.e., [1], [2]) are mentioned in the text but omitted from core evaluations.\n- The discussion on experimental results is limited. Table 1 appears to contain inconsistencies in the reported average scores. There is insufficient analysis explaining the performance trends across different methods. Notably, the HelpSteer2 RM [4] achieves higher performance than PRISM across multiple categories (also, in average score \"88.8\"), but this observation is neither highlighted nor analyzed in the text.\n- The individual contributions of kernel distance and decorrelation regularization terms remain unclear. While the authors adopt a curriculum learning schedule for the regularization weights (i.e., $\\lambda_1$ and $\\lambda_2$), a more detailed ablation study is necessary to understand their respective impacts on generalization and sensitivity to shortcut features.\n- Although the use of random feature maps offers efficiency, the cost of computing multiple embeddings, kernel, and decorrelation terms may still be significant, especially in large-scale settings. The paper lacks an analysis of the computational overhead introduced by kernel-based regularization.  While PRISM is claimed to be more efficient than ensemble-based methods, this claim is not supported by comparisons.\n\n[1] Coste et al., Reward model ensembles help mitigate overoptimization. In ICLR 2024.\n\n[2] Wang et al., Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In EMNLP 2024.\n\n[3] Hong et al., On the Robustness of Reward Models for Language Model Alignment, In ICML 2025.\n\n[4] Wang et al., Helpsteer 2: Open-source dataset for training top-performing reward models. In NeurIPS Datasets and Benchmarks Track 2024.\n\nQUESTIONS:\n- Please provide detailed discussion and comparative analysis against reward ensembles, multi-objective reward models, and last-layer retraining approaches.\n- Current presentation and discussion of experimental results lack depth and accuracy. A more substantive discussion of what drives PRISM’s performance beyond numerical gains, would greatly improve the interpretability and impact of the results.\n- Please analyze the individual and joint contributions of regularization terms (i.e., ablation study on $\\lambda_1$ and $\\lambda_2$).\n- What is the computational overhead of PRISM relative to baseline methods and computationally intensive alternatives? Please elaborate on PRISM’s scalability and its applicability to large-scale settings.\n- How does the performance degrade when shortcut distributions are unbalanced, non-compact, or noisy?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the mitigation of the shortcut phenomena in RLHF, which means the rewards learned from human preferences are correlated with  an approach for disentangling features of spurious attributes with other features in RLHF.\n\nIn addition to maximizing the reward difference between the accepted response and the rejected response, the idea is to minimizing the kernel correlation between the two responses computed using spurious attributes and the covariance between rewards and spurious attributes.\n\nThe authors provide generalization bound of their proposed kernel function and experiment results on RewardBench and RM-Bench.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n1. The idea, modeling the correlation induced by spurious features with group-invariant kernels, is well motivated and reasonable. \n2. The resulting approach looks easy-to-implement, and it is more effective than alternative methods on average.\n\n# Weakness\n1. Some minor details are missing, see questions below.\n2. Although the proposed method is claimed to reduce the correlation between rewards and spurious features, it is not supported by the Pearson analysis presented in Figure 4. In particular, the authors state that the BT model exhibits **non-trivial** correlations with tone and sycophancy, which is reduced to **near-zero** values by the proposed methods. However, for Tone, the person coefficient is reduced from r=0.019 p=0.2393 to 0.007 p=0.6378, which does not support the statement.\n\nQUESTIONS:\n1. Why does the BT model induces low correlation between rewards and tone (or sycophancy)? Is this caused by the procedure of computing feature values for tone (or sycophancy)? Could you provided results on datasets in which such correlation is confirmed?\n\n2. What is the LRU mentioned in line 230 responsible for?\n\n3. How do you encode the spurious features as vectors?\n\nLIMITATIONS:\nThere is no discussion on potential social impact. In my opinion, one potential negative impact lies in the choice of assessing values of spurious features. For example, in certain culture (e.g. Japanese), people tend to prefer very polite languages. Is there a principled approach to differentiate between sycophancy and politeness?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

comments: SUMMARY:
This paper introduces a framework that employs group-invariant kernels as a regularization technique to address shortcut learning in Bradley-Terry reward models. Through comprehensive experiments on two reward benchmarks, the authors demonstrate the efficacy of this approach. Additionally, a correlation analysis confirms the method's effectiveness in mitigating shortcuts.

STRENGTHS AND WEAKNESSES:
# Strengths
- The paper is well-structured and clearly written.
- The proposed method is supported by a theoretical guarantee, as presented in Theorem 2.
- The authors validate their approach through experiments on two standard reward learning benchmarks, demonstrating its applicability in relevant settings.


# Weaknesses
- My main concern is the significance of the problem.  In reward modeling, the critical challenge is often **identifying the shortcut**. Once a shortcut is identified, numerous methods can be employed to mitigate it. For instance, a straightforward approach is to balance the data distribution with respect to the shortcut feature. The paper would be significantly strengthened by comparing the proposed method against this simple yet powerful baseline to demonstrate its relative benefits.

- A second concern relates to the experimental results. The reported improvements over the GRM appear to be marginal. Furthermore, the paper does not include a comparison between GRM, SSRM and PRISM on the RM-Bench. Could the authors elaborate on the reason for this omission or provide the results for this comparison? This would help to better contextualize the performance of the proposed method.

QUESTIONS:
Please refer to the Strengths And Weaknesses part.

LIMITATIONS:
Please refer to the Weaknesses part.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses reward hacking in preference-based alignment by framing such failures as shortcut behaviors. Drawing on kernel-invariant theory, the authors propose PRISM, which learns group-invariant kernels via a closed-form objective so the reward model focuses on invariant, task-relevant features rather than spurious ones. Across several benchmarks, PRISM consistently raises out-of-distribution reward accuracy and, when used to train policies, lowers reliance on shortcut signals, offering a more robust foundation for RLHF.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. Reward hacking is a pressing issue in preference-based alignment, and this work offers a fundamental characterization of how it manifests and a principled solution for training more reliable reward models.\n2. The paper is well structured: the related-work section pinpoints gaps in prior research, the method is presented concisely, and a full theoretical analysis backs the approach. Experiments also highlight the method’s efficiency and scalability.\n3. The mathematical proofs are thorough and transparent, convincingly showing how the proposed invariant-kernel formulation mitigates shortcut exploitation in reward models.\n\n**Weaknesses**\n\n1. The symbols in the proof process of this paper can be slightly organized. Readers may feel that the symbols of the proof are a little messy when reading the entire proof process, but the whole process is complete and flawless.\n2. Adding a few qualitative case studies that illustrate how PRISM curbs concrete instances of reward hacking would make the empirical findings more intuitive.\n\nQUESTIONS:\n1. The symbols in the proof process of this paper can be slightly organized. Readers may feel that the symbols of the proof are a little messy when reading the entire proof process, but the whole process is complete and flawless.\n2. Adding a few qualitative case studies that illustrate how PRISM curbs concrete instances of reward hacking would make the empirical findings more intuitive.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}