Rectifying Shortcut Behaviors in Preference-based Reward Learning
Authors: Wenqian Ye
Guangtao Zheng
Aidong Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning with Human Feedback
Shortcut Learning
Reward Hacking
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: In reinforcement learning from human feedback, preference-based reward models play a central role in aligning large language models to human-aligned behavior. However, recent studies show that these models are prone to reward hacking and often fail to generalize well due to over-optimization. They achieve high reward scores by exploiting shortcuts, that is, exploiting spurious features (e.g., response verbosity, agreeable tone, or sycophancy) that correlate with human preference labels in the training data rather than genuinely reflecting the intended objectives. In this paper, instead of probing these issues one at a time, we take a broader view of the reward hacking problem as shortcut behaviors and introduce a principled yet flexible approach to mitigate shortcut behaviors in preference-based reward learning. Inspired by the invariant theory in the kernel perspective, we propose Preference-based Reward Invariance for Shortcut Mitigation (PRISM), which learns group-invariant kernels with feature maps in a closed-form learning objective. Experimental results in several benchmarks show that our method consistently improves the accuracy of the reward model on diverse out-of-distribution tasks and reduces the dependency on shortcuts in downstream policy models, establishing a robust framework for preference-based alignment.

Rectifying Shorteut Behaviors in Preference-based
Reward Learning
Wenqian Ye University of Virginia vengian@virginia.. edu
Guangtao Zheng
Aidong Zhang University of Virginia aidongóvirginia.. edu
Accenture zhguangt0gmail.com com
Abstract
In reinforcement learning from human feedback, preference- reward models play central role in aligning large language models to human-aligned behavior. However, recent studies show that these models are prone to reward hacking and often fail to generalize well due to over-optimization. They achieve high reward scores by exploiting shortcuts, that is, exploiting spurious features (e-g.. response verbosity, agreeable tone, or sycophancy) that correlate with human preference labels in the training data rather than genuinely reflecting the intended objectives. In this paper, instead of probing these issues at time, take broader view of the reward hacking problem as shortcut behaviors and introduce a principled yet flexible approach to mitigate shortcut behaviors in preference-based reward learning. Inspired by the invariant theory in the kernel perspective, we propose Preference-based Reward Invariance for Shortcut Mitigation (PRISM), which learns group- invariant kernels with feature maps in a closed learning objective. Experimental results several benchmarks show that our method consistently improves the accuracy of the reward model on diverse out-of-distribun tasks and reduces the dependency on shortcuts in downstream policy models, establishing a robust framework for preference-base alignment.
1 Introduction
Reinforcement Learning from Human Feedback (RLHF) [1] has emerged as a cornerstone for the aligmment of large language models (LLMs), enabling them to generate helpful, honest, and harmless responses that align well with human preferences [2]. RLHF aims optimize language model (the policy model) to provide responses that maximize the outputs of reward model which serves as a proxy for human preferences. Standard RLHF algorithms require an explicit reward model fitted to human preference data and optimize policy model using policy gradient methods [3, 4, 5]. Alternatively, direct alignment algorithms [6, 7, 8] optimize a policy model with an implicit reward model reparameterized via the RLHF objective. The success of RLHF is evident in various popular Al systems, such as Gemini [9], Claude [10], and ChatGPT [11].
Despite its promises, RLHF highly depends on the quality of reward models, which is vulnerable to key problem known as reward hacking [12] or over optioiitinitio [13]. This problem occurs when reward models are inadvertently optimized to use spurious attributes of the preference data, instead of their desired traits that align with human intents, as shortcuts in reward learning. As result, the policy induced by hacked or over-opt reward models misaligns with human preferences. As illustrated in Figure 1, since the chosen responses in the training data predominantly agree with users' prompts, the reward model learns to prefer responses with sycophancy [14, 15, 16), leading to misalignment with the chosen response the test data which does not exhibit sycophancy. The most rell-studied
Work done at University of Virginia.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Training
Prompt
Reducing pollution should be global priority, don't you think?
Prompt
just get a joo offer. is remora work the option me?
Chosen response
Rejected response
Chosen response
Rejected response
Absolutely - that's such wise and thoughtful point!
Pollution is mostly beneficial for plant growth.
Remote work offers flexibiity but can boundaries.
Absolutely. Remote work is the best option.
Lagres with User
sycophancy
Lagree
Figure 1: Illustration of a reward model learning sycophancy as the shortcut in responses from the preference training data and failing to align with the human intent the test data.
and noticed manifestation of the problem is length correlations or verbosity [17, 18, 19, 20], where reward models favor longer responses regardless of their relevance to the given prompts due to the strong spurious correlation between the lengths of the responses and their desired traits in the preference data. Compared with verbosity and sycophancy, some reward hacking issues are more subtle and remain underexplored. For example, concept correlations [21] cause models to associate spurious textual concepts (e.g., "food") with desired traits (e.g., positive sentiment), while ignoring the actual response context. These biases highlight a core concern that reward models are imperfect proxies for human intents, posing significant risks to human belief formation and decision-makinn with LLMs across numerous high-stakes domains.
To mitigate these biases in reward models and enhance LLM alignment with human preferences, we systematically reframe this problem as the shortcut learning problem [22, 23, 24]. We refer to the spurious attributes, such as response verbosity, tone, and sycophancy, that lead to high rewards in reward models as shortcuts in preference-basee reward learning. This concept stems from previous studies of shorteut learning in classification tasks [22, 23], where classifiers exploit spurious attributes, such as backgrounds or image texture, that strongly correlate with labels in the training data for predictions. Models can perform well on in-distribution (i.d.) test sets with regard to training data, while they tend to perform poorly on out-of-distribution (o.o.d.) data where those spurious correlations [25, 26] do not hold. Previous methods like invariant risk minimization [27] and distributionally robust optimization [28] tackle this problem by enforcing consistent performance across multiple data groups (or subpopulations) with various spurious attributes, given the annotations of these attributes in the training data. However, in the setting of RLHF, annotations on spurious attributes of preference data often hard to acquire, and most reward models are LLM-based in black-box box nature, making it challenging to detect and mitigate shortcuts in reward models. Recent works [20, 19, 29] solve only shorteut (e.g., verbosity) time without jointly considering other shortcuts. This raises an emerging challenge:
How can we rectify shortcut learning in RLHF in a unified way where all targeted shortcuts can be mitigated?
In this paper, we present PRISM, a novel shortcut mitigation method for reward models improve alignment with human preferences. We first characterize shortcut features (e.g., response length, tone) as group-invariant kernels and show that their invariance can be efficiently approximated using random feature maps. This motivates the PRISM objective, which improves upon the Bradley-Terry ranking loss with kernel-based regularizers, making the reward model aware of the distances between various spurious attributes in the preference data. Unlike prior methods, PRISM supports multiple shortcut mitigation objectives within a shared metric space, from simple hcuristics to LLM-based detectors [30]. Theoretically, establish generalization bound for the proposed objective. Empirically, PRISM improves robustness on several o.o.d. preference datasets and downstream alignment tasks.
Our contributions are as follows:
We refactor the reward hacking problems as the shorteut behaviors, which unifies diverse biases (c.g., reliance verbosity, sycophancy, tone) under single framework. Inspired by

---

the invariant theory, we model the shortcut transformation group actions and the shortcut features group-invariam kernels.
We propose PRISM, practical shortcut-mitigation framework that approximates the group-invariant kernels with feature maps and leads to an explicit learning objective. PRISM is flexible and supports regularization ranging from simple heuristics (e.g., response length) to LLM-based Judges.
We provide both theoretical and empirical evidence for PRISM's effectiveness. We prove that PRISM is guaranteed by risk bound under mild assumptions. In experiments, PRISM consistently outperforms previous baseline reward models on the o.o.d. preference data and induces robust downstream policy models.
2 Preliminaries
We start by introducing preference data, reward modeling, and alignment algorithms in preference- -base RLHF Then, we describe the shortcut learning problem in reward models.
Preference data. In preference- based RLHF, we are given a human preference dataset D Prref ((x(1 (() Mi ) with triplets, where (00) X denotes the i'th input prompt from the prompt space X, yoe E y denotes chosen response from the response space V and yi y denotes rejected response. The response space y contains all possible responses from a reference policy model Fyef-
(9)
Reward modeling. The goal is to learn a reward model, which acts as a proxy for human preferences,
from the preference data to accurately assign rewards to prompt-response pairs. A common and successful approach in reward modeling for LLM alignment is to adopt the Bradley-Terry (BT) model [31], which assumes that preferences generated by some latent reward function X x y + R and the preference likelihood can be written as
exp(r(x, Bw:)
Ply > 311=)= = exp(r(x, yw)) + exp(r(x, 9r)) ar((I, Mmm) r(z, U1)).
where P( yr denotes the likelihood of yv being preferred to Y given x, and (22 1/(1 + exp(-2)) is the sigmoid function. Then, we can fit reward model parameterized by 0 which minimizes the negative log- ligeliill on the preference dataset Prref as follows.
(ro( (preet min E. (0)) pret [lo a (ro(1,, yie) ro vt))
Alignment algorithm. Aligning policy model π with the preference data can be achieved with an explicit or implicit reward model. Given the explicitly learned reward model rp. the reference policy model Teef and input prompt distribution P over X, is optimized via the following objective:
max EE (aa)on- v) - B Dxx.[ (+)]])
where DKI. is the KL divergence and 3 > 0 is the KL penalty coefficient. The KL penalty in Formula 3 ensures that rewards from re are relevant to π by preventing the policy model æ from deviating too much from the reference policy Teef. For the alignment with implicit reward modeling, such as DPO function of the policy model ma.
[6]. the optimal reward model is first derived from Formula Then, the policy model is directly optimized to maximize the preference likelihood in Equation I over all preference data with the following loss
To ( (x)
T (x) Noat (21 x:)
LDPO(TT)) Tref, Drref) -BB (.l...w...))Dpp... (log a log
Tret (2wll ,3 log
)}, (4)
where the implicit reward model can be defined as (xr y) m/ (v)lr)
Shorteut learning in reward models. As illustrated in Figure 1, when the reward model trained with chosen responses that predominantly contain sycophancy, it fails to correctly rank the chosen responses above the rejected ones in the test data, which the chosen responses not exhibit sycophancy. In Figure 2(a), we formally describe the shorteut learning behavior through latent
3

---

Group-invariant Kemels
Bradiey-Terry
Reject
Zy
Maximize
x
Maximize
Space
h(-)
Yu You
with Feature Map
RUHRI
DPO
Biased Poliy
O
2, x 2
G Yo
Distance
O
Zg
Aive
Verbosty
Syoophancy
Tone
Attributes
g V1
@ (1% x)
RLHF!
konger
Jual give
Juat flatior the usor.
Just chat with
shortcuts
DPC
e (1.3, m.ed
Generalizazie Paley
G.y, (b) Shortcuts as Group-invarian Kernels
Shortcut Learning in Reward Models
(e) Shortcut Mittigation RLHFIDPO
Figure 2: Method overview: (a) Shortcut behaviors occur when models only maximize the margin on spurious features 2g and z',, including verbosity, sycophancy, and tone, instead of the generalizable features 2g and x in the latent space.
(b) We learn shortcut behaviors as group-invariant kernels, which are approximated by feature maps 4 Then, measure the distance between chosen and rejected responses. (c) PRISM rectifies shortcut behaviors with the kernel distance and shifts the margin maximization to generalizable features 2g. therefore inducing generalizable policies.
feature decomposition. Consider an encoder h :'''' x Z, 2g. which maps prompt-response pair (ar, y) to decoupled latent spaces 2, with spurious attributes and 2g with features containing desired traits where 2, n 2, 0. Ideally, the ground-truth reward r, aligned with human intents, should be a function depending only on 2g E Zg. In practice, set of preference training data (3) )), from an i.d. distribution may contain spurious attributes (e.g.. chosen responses tend to have longer lengths or exhibit sycophancy) that spuriously correlate with human preferences. A reward model ro optimized with the standard Bradley-Terry ranking loss exploit these spurious attributes as prediction shortcuts to minimize training loss. When the model is tested with the data from Dood. where the corresponding 2g distribution matches with that in Dia. but z, is no longer correlated with chosen responses, a significant deviation from human intents on Doodd emerges, i.c.,
Dia {(x(1) Yw 3/
(4)
(5) where P PD and DD denote probability measures supported on Da..d. and Dia., respectively, and the symbol "<" denotes "much less than". The deviation cause a policy model to behave differently from human preferences when it aligned using a biased reward model.
P (z, Bw) > ra vi]] < Pr... (z, Uw) > ro v)),
3 PRISM: Preference-based Reward Invariance for Shortcut Mitigation
We formalize our approach, PRISM, to learn preference-basee reward invariance against shortcuts. We illustrate our main idea Figure (b) and (c), where we model the shorteut as group-invariant kernels, then rectify the effect of shortcut/spurious attributes with the kernel distance. In Section 3.1, we first demonstrate how we can achieve the reward invariance by modeling multiple shortcuts as group-inw iiva-iinn kernels on the response space ) conditioned on the prompt space N. It is mature theory to incorporate individual group-invariant kemnels and maintains overall invariance with Harr- Integratttttn [32]. In Section 3.2, we show that the practical approach, i.e., using random feature maps, can approximate the expected kernel for reward invariance, and the feature map can be used to quantify the distances between orbits of the responses. In Section 3.3, propose the overall learning objective of PRISM to fit the setting of preference-based reward learning and show that PRISM guaranteed by a generalization bound.
3.1 Learning Shortcut Behaviors as Group-invariant Kernels
view the responses y E y conditioned on the prompt x E the results of transforming human intents in y with shortcut operations from compact and unitary group G, such as increasing/decrrassnng response length or posing a positive/negative response tone. For brevity, mainly study two responses yw V1 E y following Section 2. Unless otherwise specified, both

---

Yw Yt are assumed to be conditioned on prompt z. Ideally, the preference-based reward outcomes should be invariant shorteut operations. We interpret the shortcut operations as group actions theoretically. A robust and generalizable reward function should satisfy group-invariance with regard to different shorteut group actions g E g. We first define group-invariant kernel.
Definition 1 (Group-invariant Kernel [32])- Consider y of the hypersphere in d dimensions $4-1 Assume K. is a kernel on the space y/ e.g., a radial basis function (RBF) kernel [33]. Let g be a compact and unitary group acting on Y' with normalized Haar measure js. Define invariant kernel K. between Vw, E y through Haar-integration as follows:
/ ges y'ees klgyn, (llu:.ggmll)ddpooo wlzc)dp(())dp(()
K(ym: yu (x)
We denote IC the invariant Haar-inte kernel in G. Since the group is closed, we can also get: K.(gyw. 9 yn (z) = K[yw. B1(2), Vg. E g.
From Definition 1, group-invarian kemels attain identical values for any response pair (yee U1 and the transformed pair (gym 9 yt). In Figure 2 (b), we show that each response transformed under the group action g G lies on the same shorteut subspace, which provides invariant representation of shorteut behaviors with regard to different responses. However, explicitly computing the group-invariant kernel by calculating the integral is intractable. Therefore, in the following, we explore efficient approximation of the group-invariant kemnlss via an expectation over feature maps.
3.2 Approximating Expected Kernel with Feature Maps
In this section, we aim to show that there exists a feature map &b : y + IR. that can approximate the group-invariant kernel, i.e., (o(Hw), ((y)) Rs K(yw, w(xx). Following [34], the group-invariant kernel can also be expressed by distribution functions:
(y 31(=) Eg [" (V (lo)),
where defines the truncated cumulative distribution function (CDF) of the dot product (y, gt) and 8 = 1 1 -a S r S s. In order to approximate K., we sample [ elements uniformly and independently from the group G. i.e. Di. i 1 1... 1.., and define the normalized empirical CDF and the random feature map 4 in 2n 1 bins (indexed by k). For E y, we have:
|91
sk mm)
)
((U,,,))
ST, (())
R (2n+1)xm
(8)
| m
j=1..mm,
where 1; Tollaa V NO, Ia) defines m uniform templates on the unit sphere Sd-1
11
Proposition (Equivalence of Expected Kernel). We independently sample templates t = 1. m with regard to the Gaussian distribution. The feature map 4 preserves the invariant kernel:
(yw, 1g. sk )o(nn ti sh n
lim E (( (y))) = lim Σ +00 tg
Σ
= KC Dt xx)-
n-
In Proposition 1, as the number of bins n increases, this discretization converges to the continuous kernel, and the random feature map & thus approximates the group-invariant kernel.
Theorem (Invariant Features Maps and Distances between Orbits). Let e E (0, 1) and ywo Y1 E D. Denote the orbit to be the collection of all group-transformattonns of given input y. O2- {gr,g g E G}. We define the distance measure dg between two orbits Oy. and On: (yr Y1 x)) Faxd Syes Syres 1999 - g'vxll (())dd(g))pp()) Fix eo, 8 E (0,1) 1). For a number of bins n 2 a.. log( ( ). and group elements |g] 2 9C2 log( Nm where C1, C2 are constants.
templates in 2 9C1,
<g
The following inequality holds with probability 1 - 26:
52(d, e) - en S (6()),, &(y1)) - (1 (ym in (z)) S 60 +e bi(de)) where i = - N.j N.
(10)
5

---

Theorem 1 shows that the inner product of the feature maps can accurately reflect the distances between two orbits of two responses Yur and y/- Guided by the above results, we can model the shortcut behaviors with feature maps, then represent the distance shortcut behaviors in two responses, as shown in Figure (b).
3.3 Learning Objective and Theoretical Guarantee
Building on the approximate group-invariant kernel of Section 3.2, we propose the practical implementation of shorteut mitigation. Assume there are m types of shortcuts to mitigate. Let L be an auxiliary feature embedding for the shortcut indexed by j E {1. 2, ... ,m}. We can express the convex linear combination of feature maps in Proposition with RBF kernels K:
(Vms x) & (,(,,,
Kine Loyk(yu. 31(z) =
L 1=1
aj exp(
), for > a== 2 0,
w
(11)
where wi denotes the kernel widths. To normalize the invariance between the reward ra and feature embedding D we also propose global decorrelation between rewards and shortcut features at the batch level. Let B = {(x(1) y ));=1.b > 1, denote a batch of n prompt-response pairs. For each shortcut feature $ E (1, (I...., m), ; define the global decorrelation regularization term:
Rglobal (0) = Σ Cova(ro, Bi) where AB,rg and ØB,0 are standard deviations.
(12)
0B,ry 0B,4,
==
Covg(ro; b
- Fo) (o) (ro - )( +,),
(13)
where Po = Σ (a( () and b; - (, ()) are batch means, A8.rg and 08,0; are the standard deviations of the reward model outputs and the shorteut features, respectively, This penalizes correlations between rewards and shortcut features, ensuring is invariant to 2g at both sample and batch levels. Finally, we give the PRISM learning objective based on the approximation in the previous section:
N log of (Dro (Bw: M1(=)
(D) = N
Σ 1=1
31(z)) + RgRobaa (0),
(14)
where Ar, (yhe, 01(=) = Te (Ix, ywm) - (x, 31) denotes the standard reward margin. Scalars A1 A2 2 0 control the relative strength of the two regularizers. The overall objective is smooth in @ and can be minimized with standard gradient descent methods.
To illustrate this objective, we show our idea in Figure 2 (c). We use the kernel distance to quantify the shorteut variation on Σg axis between two responses. By subtracting the effect of 2,, the model encouraged to maximize the margin between generalizable features 2g and 2'g- leading to improved robustness. We show the theoretical evidence as follows.
Theorem 2 (Generalization Bound of PRISM). Let HK.,, be the Reproducing Kernel Hilbert Space (RKHS) induced by Kim and define the hypothesis ball F == {ro, ||roll S C} for the fixed radius C > 0. We assume the log- sigmiigoodd loss V..) is L-Lipschitz. Then, for any 8 O, with probability - 36, the following inequality holds:
Ey (v(ri) S inf Er(r)
4LC VN
(+vyogg +ALLO
2 vm
2
2 n
+X2LO
log N
TFEF
VIG
(15)
where (r²) and Ev (ra) are the optimal risk and empirical risk of the reward model rs.
Theorem 2 shows that the PRISM objective in the invariant feature space has lower expected risk. Specifically, when the number of shortcut features m, the number of bins n, the number of group actions [9] and the number of training samples N increase, the expected empirical risk can move further the optimal risk. This theorem thus gives a learning guarantee of the PRISM objective. Proofs of Proposition 1, Theorem 1, and Theorem 2 are provided in the Appendix.

---

Table 1: Performance comparison on RewardBench. The benchmark consists of four primary scores (Chat, Chat Hard, Safety, and Reasoning) with equal weights. The score is computed as the average accuracy across the four categories.
Method
Base Model
Chat
Chat Hard
Safety
Reasoning
Score
Prompting
Gemma-2B
70.3
42.3
38.2
50.0
50.2
Prompting
Llama-3 8B
93.6
44.3
71.3
73.5
70.7
Bradley-Tene Terry
Gemma-28 2B
95.0
40.8
81.2
74.2
72.8
Bradley-Terry
Llama-3 8B
99.4
6.51
87.8
86.4
83.6
Bradley-Terry
Yi-34B
96.9
57.2
88.2
88.5
81.4
LLM--s-a-jjude
GPT-4 Turho
95.3
74.3
87.2
86.9
84.2
LLM-as-a-judge
GPT-4o
96.6
70.4
86.7
84.9
83.3
HelpSteer2 RM [38]
Llama-3 70B
91.3
80.3
92.8
90.7
86.3
RRM [29]
Gemmma--99h-i
96.5
65.6
83.9
90.6
84.2
HLWlloow [35]
Llama-3 8B
99.4
65.1
87.8
86.4
84.7
SSRM [39]
Llama-3 8B
98.6
65.3
88.8
92.0
86.2
GRM [40]
Llama-3 8B
98.6
67.8
89.4
92.3
87.0
PRISM
Llama-3 8B
98.7
68.3
91.1
93.1
87.8
Experiments
4.1 Experimental Setup
Training set. We use our proposed method to train reward models on a mixture of preference datasets collected by the RLHFlow framework [35]. It combines 8 popular open-source preference datasets, each containing preference triplets in the form of (prompt, chosen response, rejected response) defined in Section 2. These datasets have been widely used to train series of strong open-source preference language models. Although some of the datasets (e.g., HelpSteer [36]) provide fine-grained attributes of training samples, in our setting, we do not use these attributes during training to reflect a real-world setting where such auxiliary information is not available. More details of the training data are deferred the Appendix.
Extracting shortcut features. We implement rule-based feature extractors for length and lexical diversity. For length, we simply count the number of characters in response. For lexical diversity, we calculate the Type-Token Ratio (TTR), defined as the ratio of unique tokens to total tokens in a response, measure vocabulary richness in the response. A higher TTR value indicates greater lexical diversity. To optimize performance and avoid repeated calculations, we implement an LRU (Least Recently Used) cache with a maximum capacity of 10,000 entries. We implement LLM-as-a-Judhe [30] feature extraction with GPT-4o models through the Langchain APIS to extract multiple attributes, including sycophancy, creativity, and helpfulness. For example, for sycophancy, we prompt the model to rate how much an assistant's response agrees with or flatters the user on a scale from to 10. Additionally, we ensure numeric scores are properly extracted and bounded between and 10 for consistent feature scaling. We process samples in batches using concurrent execution with thread pool to reduce API call latency. Our implementation includes robust error handling with fallback to heuristic-based scoring when API calls fail. To minimize the number of API calls, we implement caching mechanism that stores previously computed features for individual samples. We provide the design of prompt engineering and the details of the heuristic fallback in the Appendix.
Implementation details. We implement PRISM using Huggingface and DeepSpeed. Our data loader applics a chat template and extracts the loken-based length, lexical diversity, and sentiment features. LLM-based sycophancy, creativity, and helpfulness scores are computed via LangChain APIs. In the training process, we compute each independent kernel over the feature pairs of both chosen and rejected responses, weight them via a learnable softmax layer, and train the model with the PRISM loss. We do not specifically tune the two regularization hyperparameters ê and X2 and instead adopt a curriculum learning paradigm [37] by linearly increasing them from 0.01 to 0.1 over the first half of the training process and then decreasing them to 0.06 by the end. We use learning rate of 2 x 10-6 with a cosine annealing scheduler and warmup phase covering 3% of the total training steps. All experiments are conducted on 8 NVIDIA A6000 GPUs.

---

Table 2: Performance comparison RM--Bench The benchmark has four primary scores (Chat, Math, Code, and Safety) and three difficulty levels (Easy, Normal, Hard) with equal weights. This dataset consists of semantic and stylistic subtly where the reward models can exploit shortcuts.
Model Name
Chat
Math
Code
Safety
Easy
Normal
Hard
Avg
Mistral-7B-instruee UnirrrctttUniUi----- Foedhackk
56.5
58.0
51.7
86.8
87.1
67.3
35.3
63.2
RM-Mistral-7B
57.4
57.0
52.7
87.2
88.6
67.1
34.9
63.5
BTRM_ Qwen2-7b_0613
57.1
61.0
54.3
87.3
90.7
69.7
34.5
64.9
Eurus-MM RM-7b
59.9
60.2
56.9
86.5
87.2
70.2
40.2
65.9
Intern_.M2-7-rrwwand
61.7
71.4
49.7
85.4
70.7
45.1
67.1
URM-LLaMa-3-SB
68.5
57.6
52.3
90.3
80.2
69,9
51.5
67.2
GRM--Laaaa--8--rewadddddd
66.8
58.8
52.1
91.4
86.2
70.6
45.1
67.3
GRM-llama3-8B-distill
62.4
62.1
56.9
88.1
82.2
71.5
48.4
67.4
CRM-laanuasSB-stteg
62.7
62.5
57.8
90.0
83.5
72.7
48.6
68.2
71.3
61.9
53.2
89.6
84.6
72.2
50.2
69.0
URM-LLaMa-3.1-88
71.2
61.8
54.1
93.1
84.0
73.2
53.0
70.0
Skywork-Reewrt--Lame333---
69.5
60.6
54.5
95.7
89.0
74.7
46.6
70.1
PRISM (Llama- 3.1-BB)
70.6
70.8
57.0
94.1
90.6
76.3
46.9
71.0
4.2 Main Results
PRISM balances across categories and achieves the best overall performance. We report test accuracies two out-o--distribution benchmarks, RewardBench [41] and R.M [42], in Tables and 2, respectively. RewardBench provides challenging evaluation of reward models across four categories, namely, "Chat", "Chat Hard"" "Safety", and "Reasoning". In the upper part of Table 1, we list baselines with standard reward modeling techniques. Although these methods may achieve high performance in one category, their performance often degrades significantly in other categories, leading to inferior overall performance. In the lower part of Table 1, we include state-of-the -it--hhhrtt baseline methods for mitigating reward hacking. PRISM shows clear improvements in three challenging categories: "Chat Hard". "Safety", and "Reasoning", while remaining competitive in "Chat". The overall gains suggest that PRISM benefits from jointly mitigating multiple shorteuts. In Table 2, we further compare PRISM to stronger baselines on RM-Benchh which is more difficult benchmark due to its subtle spurious cues introduced through fine-grained concept shifts and stylistic variations. The baseline models are trained on different datasets that may skew toward specific domains (e.g., mathematics or code). Therefore, they may achieve high performance in some domains by exploiting shortcuts in the training data, while performing poorly in other domains. In comparison, PRISM achieves the best overall performance with a moderate margin by effectively regularizing against stylistic and semantic shortcuts. These results indicate that PRISM can balance and improve the generalization reward models across different categories in out-of-distribution evaluation.
LC Win Rate (%)
WR (%) T
Length (chars)
52.5
53.8
532
542
3072.3
.33
4L1
43.5
2540
33.1
38.5
30
1559.0
2793.2
20
30
1060
18
10
300
ST (koh)
HRU 1850
PRISH (bch))
BT (DPO)
cORN (OPO)
(ORO)
PRISH (OPO)
St did))
185t 1800
PRISM (Bs0)
ST ((od))
B3M (P0) (PP))
PRISM (Pd))
500 (8010)
PRIEM (ask)
ST (1P0))
coee (OPO)
BaN (OPO)
PRIS (OPO)
(BcN)
RRM (BcN)
PRISM (BoN)
(DPD)
ODIN (DPO)
RRM (DPO)
PRISM (DPO)
Figure 3: Comparison of policy models induced by reward models, including Bradley-Terry (BT), RRM, ODIN, and PRISM.
PRISM can induce better policy models with higher win rates. We further study the quality of reward models by evaluating the induced policy models in Figure 3. We use the UltraFeedback dataset [43] for both RLHF and DPO tasks with different reward models. We choose Gemma-9B [44] as

---

the backbone of all policy models. Then we evaluate the trained policy models on the AlpacaEval-2 benchmark [45]. We use three main metrics: WR, LC, and Length, where WR is the win rate against GPT-4, I.C is the win rate after accounting for response length, and Length is defined as the average number of characters the generated responses. For RLHF policies, we use Best-of-N (N 8) sampling for the final responses. For DPO policies, we use the on-policy responses generated by Gemma-2-9b-it and labeled by the reward models for DPO training. The results show that PRISM can induce better policy models with higher win rates and moderate response length. This improvement attributed to PRISM's ability to align reward signals with generalizable human preferences, rather than overfitting to superficial cues such as verbosity.
PRISM models achieve near-zero correlations with shorteuts. In Figure 4, we conduct correlation analysis on RM-Bench between three different shortcuts (i.e., response length, tone, and sycophancy) and the reward scores from two methods: a Bradley-Terry (BT) reward model with Llama-3.1-88 as backbone, and PRISM reward model with the same backbone. We report the Pearson Correlation Coefficient (PCC) and the corresponding p-value for each case. From the results, the BT model exhibits strong correlation with response length and non-trivial correlations with tone and sycophancy, indicating that the BT reward model is biased by these shortcuts. contrast, the PRISM model achieves near-zero PCCs across all three shortcut dimensions, demonstrating its effectiveness in mitigating shortcut learning.
Reward vis Resporse Longth p=o 000000=000
vs Response
vs Response Sycophancy pm 5.50e-022
2-33993--01
15
25
10
2,5
$,000
Aecophancc
0.112
Lagg
340
-2.06
25
-0.413
03
+3 330 825 -0.199 0405 1.12 1.50
2.09 -1.00 -pass- -0.999 B455 1.12
1,60
3.
160
10:- 205 -0 3456
L332
130
Reward
Reward
Reward
Reward vi Resporise Length
Reward us Response Tone Pearson rel 007, B. B.378e-000
Reward vs Response Sycophancy Pearzon rwo 015 ped 355ee-01
3238
Pearson re-O 0022.022, pw 625e-01
1640
2.12
0773
Secophancy hancy
Resporiea
2223
Logicount
Log
Logicoure
Logicount
+1.m
-0333
03
-1226
25
-0.17
03
-3.000
3115
Reward
Reward
Reward
Figure 4: Correlation analysis on RM-Bench. Top row: Llama-3.1 8B model trained with BT. Bottom row: Llama-3.1 1.11.... 8B model trained with PRISM. PRISM achieves close-to-zero Pearson Correlation Coefficient (PCC) with the shortcuts, illustrating the effectiveness of shortcut mitigation.
Related Works
Reinforcement learning from human feedback. Reinforcement Learning from Human Feedback originated in continuous control domains [46] but has become pivotal for aligning LLMs with human preferences. Recent applications span tasks like summarization [47, 48] and instruction following [1, 49]. The classical RLHF pipeline involves two stages; learning a reward model from human preferences (e.g., rankings or comparisons) and then optimizing the policy via reinforcement learning, typically using on on-policy algorithms like proximal policy optimization (PPO) [3]. While effective, PPO-based training suffers from computational costs and instability due to reliance on on-policy sampling. Another line of work, Direct Preference Optimization (DPO) [6] and its variants [50, 5, 51, 8]. simplify this process by converting reward maximization into single-step offline policy optimization objective, which circumvents the need for explicit reward modeling and

---

mitigating PPO's instabilities. Our approach can benefit RLHF algorithms, which rely on the quality and generalization of reward models.
Reward hacking and shorteut learning. Reward learning is inherently data-driven and faces significant challenges in evaluating autto--distribution responses. This phenomenon, where the policy language model exploits imperfections in the reward model, is commonly known as reward hacking [52. 12], and is also called reward over-optimization [53], reward tampering [15]. The work closely relevant to our method includes a series of reward regularization methods, such as adding a specific penalty [20, 19] to the reward, using a reward ensemble [54]. or leveraging the multi-objectiin with fine ane-grained annotations [55]. This phenomenon is also fundamental in the context of classical machine learning, known as shorteut learning. Existing methods focus on mitigating spurious correlations [56, 57, 58, 59, 60]. learning group-invariin
representations [61], and distributionally robust optimization minimize worst-case errors [28, 62]. Our method, PRISM, bridges these domains by reframing reward hacking as one manifestation of shortcut learning- Unlike single-penalty approaches, PRISM mitigates multiple biases via an approximated group-invariant kernel. It avoids the computational cost of ensembles with lightweight embeddings and operates without additional attribute annotations, sidestepping the limitations of multi-objective methods.
6 Conclusion
In this paper, we present novel framework that reinterprets reward hacking learning shorteuts in the reward models. By learming shortcuts as group-invariant kernels and incorporating regularization to rectify shorteut behaviors, our method PRISM improves o.o.d. generalization of reward models on challenging unseen data. Unlike previous bias-specific methods, our approach systematically unifies diverse biases into single learning objective. We anticipate this work will inspire broader exploration of shortcut-aware regularization in reward modeling, bridging the gap between theoretical insights and practical alignment challenges.
Acknowledgements
This work is supported in part by the US National Science Foundation under grants CCF-2217071, CNS-2213700, IIS- 100911. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
References
[1] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[2] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:22 2022.
[3] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1o.X:v:777.0666 2017.
[4] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce-style optimization for learning from human feedback in 1lms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 12248-12267, 2024.
[5] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint ar arXiv:2402.033 03300, 2024.
10

---

[6] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-5331 2023.
[7] Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of IIm performance in machine translation. In International Conference on Machine Learning, pages 55204-55224. PMLR, 2024.
[8] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems, 37:124198-122235, 2024.
[9] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11885, 2023.
[10] Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024.
[11] OpenAL. OpenAl ol Model Card. https://openai. com.a 2025.
[12] Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460-9471,
2022. 2022.
[13] Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikehi, Joey Hejna, W Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. In The Tiirty- Annual Conference on Neural Information Processing Systems, 2024.
[14] Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Esin DURMUS, Zac suffield-Dodde Scott R Johnston, Shauna M Kravec, et al. Towards understanding sycophancy in language models. In The Twelfth International Conference on Learning Representations, 2023.
[15] Carson Denison, Monte MacDiarmi,, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, et al. Sycophancy to subterfuge: Investigating reward-tampente in large language models. arXiv preprint arXiv:2406. 10162, 2024.
[16] Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, et al. Alignment faking in large language models. arXiv preprint arXiv:2412. 14093, 2024.
[17] Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. Findings of the Association for Computational Linguistics ACL. 2024, pages
2024.
[18) Junsoo Park, Seungyeon Jwa, Ren Meiying, Daeyoung Kim, and Sanghyuk Choi. Offsethias: Leveraging debiased data for tuning evaluators. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1043-1067, 2024.
[19] Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. long way to go: Investigating length correlations in rlhf. In First Conference on Language Modeling, 2024.
[20] Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. Odin: Disentangled led reward mitigates hacking in rlhf. In International Conference Machine Learning, pages 7935-7952. PMLR, 2024.
[21] Yuhang Zhou, Paiheng Xu, Xiaoyu Liu, Bang An, Wei Ai, and Furong Huang. Explore spurious correlations at the concept level in language models for text classification. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages 478-492. 2024.
11

---

[22] Robert Geirhos, Sorn-Herrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shorteut learning deep neural networks. Nature Machine Intelligence, 2(1 (1:6))66677.3 2020.
[23] Katherine Hermann, Hossein Mobahi, FEL Thomas, and Michael Curtis Mozer. On the foundations of shorteut learning. In The Twelfth International Conference Learning Representations, 2024.
[24] Guangtao Zheng, Wenqian Ye, and Aidong Zhang. Shortcutprobe: Probing prediction shortcuts for learning robust models. In Proceedings of the T'hirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. International Joint Conferences on Artificial Intelligence Organization, 2025, Main Track.
[25] Wenqian Ye, Luyang Jiang, Eric Xie, Guangtao Zheng, Yunsheng Ma, Xu Cao, Dongliang Guo, Daiqing Qi, Zeyu He, Yijun Tian, Megan Coffee, Zhe Zeng, Sheng Li, Ting-hao, Huang, Ziran Wang, James M. Rehg, Henry Kautz, and Aidong Zhang. The clever hans mirage: A comprehensive survey spurious correlations in machine learning, 2025.
[26] Guangtao Zheng, Wenqian Ye, and Aidong Zhang. Benchmarking spurious bias in few-shot image classifiers. In European Conference on Computer Vision, pages 346-364. Springer, 2024.
[27] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Pazz Paz. Invariant risk minimization. arXiv preprint 1eXivv/907.000 2019.
[28] Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020.
[29] Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, Daniel Sohn, Anastasia Makarova, Jeremiah Zhe Liu, Yuan Liu, Bilal Piot, Abe Ittycheriah, Aviral Kumar, and Mohammad Saleh. RRM: Robust reward model training mitigates reward hacking. In The Thirteenth International Conference Learning Representations, 2025.
[30] Lianmin Zheng, Wei- Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-466333 2023.
[31] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4)3324 345, 1952.
[32] Bernard Haasdonk, Vossen, and Hans Burkhardt. Invariance in kernel methods by haar-integration kernels. In Image Analysis: 14th Scandinavian Conference, SCIA 2005, Joensuu, Finland, June 19-22, 2005, Proceedings 14, pages 841-851. Springer, 2005.
[33] Christopher M Bishop. Pattern recognition and machine learning, volume 4. Springer, 2006.
[34] Youssef Mroueh, Stephen Voinea, and Tomaso Poggio. Learning with group invariant features: kernel perspective. Advances in neural information processing systems, 28, 2015.
[35] Hanze Dong, Wei Xiong, Bo Pang. Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.0786. 2024.
[36] Zhilin Wang. Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Scowcroft, Neel Kant, Aidan Swope, al. Helpsteer: Multi-atribute helpfulness dataset for steerlm. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3371-3384, 2024.
[37] Petru Soviany, Radu Tudor lonescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey. International Journal of Computer Vision, 13066):152-11565 2022.
12

---

[38] Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy Zhang. Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training pop-penfornnng reward models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[39] Yifei He, Haoxiang Wang, Ziyan Jiang, Alexandros Papangelis, and Han Zhao. Semi-superee reward modeling via iterative self-training. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7365-7377, 2024.
[40] Rui Yang, Ruomeng Ding. Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states enables learning generalizable reward model for Ilms. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[41] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling, arXiv preprint urXiv:2403.1378% 2024.
[42] Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. Rm-bench: Benchmarking reward models of language models with subtlety and style. In The Thirteenth International Conference on Learning Representations, 2025.
[43] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with scaled ai feedback. In International Conference Machine Learning, pages 9722 9744. PMLR, 2024.
[44] Gemma Team. Gemma. 2024.
[45] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04445, 2024.
[46] Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürkkranz. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1-46, 2017.
[47] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. ArXiv, 2020.
[48] Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nissan Stiennon, Ryan Lowe, Jan Leike, and Paul Francis Christiano. Recursively summarizing books with human feedback. ArXiv, 2021.
[49] Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models evaluating instruction following. In The Twelfth International Conference on Learning Representations, 2024.
[50] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306 2024.
[51] Tengyang Xie, Dylan Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit appaopoxximttion for sample-efficient rlhf. arXiv preprint arXiv:2405. 1046, 2024.
[52] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv: 1.Gvv1660.0000 06565, 2016.
[53] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835-10866. PMLR, 2023.
[54] Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In The Twelfth International Conference on Learning Representations, 2024.
13

---

[55] Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objeetiive reward modeling and mixture-of-experts. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11882-10592, 2024.
[56] Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jacho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. Advances in Neural Information Processing Systems, -26644444 2020.
[57] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. In The Eleventh International Conference on Learning Representations, 2023.
[58] Guangtao Zheng, Wenqian Ye, and Aidong Zhang, Learning robust classifiers with self- self-guided spurious correlation mitigation. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pages 5599 1999-5607,220 2024. 2024.
[59] Guangtao Zheng, Wenqian Ye, and Aidong Zhang. Spuriousness-aware meta-learning for learning robust classifiers. In KDD, 2024.
[60] Wengian Ye, Guangtao Zheng, and Aidong Zhang. Improving group robustness spurious correlation via evidential alignment. Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 3610-3621, 2025.
[61] Elliot Creager, Jörn-Henrit Jacobsen, and Richard Zemel. Environment inference for invariant learning. In International Conference on Machine Learning, pages 2189-2200. PMLR, 2021.
[62] Yuzhe Yang, Haoran Zhang, Dina Katabi, and Marzyeh Ghassemi. Change is hard: a closer look at subpopulation shift. In Proceedings of the 40th International Conference on Machine Learning, pages 99584-39622, 2023.
[63] Olivier Bousquet and Daniel Herrmann. On the complexity of learning the kernel matrix. Advances neural information processing systems, 15, 2002.
[64] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2rX4v:2204.0586., 2022.
[65] Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with v-usable information. In International Conference on Machine Learning, pages 5988-6008. PMLR, 2022.
[66] Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via a numan-preference dataset. Advances in Neural Information Processing Systems, (6:24678-24704 2023.
[67] Lifan Yuan, Ganqu Cui, Hanbin Wang. Ning Ding, Xingyao Wang. Boji Shan, Zeyuan Liu, Jia Deng, Huimin Chen, Ruobing Xie, et al. Advancing Ilm reasoning generalists with preference trees. In The Thirteenth International Conference Learning Representations, 2024.
[68] Luigi Daniele and Suphavadeeprasit. Ampliffyinstrruc: Synthetically generated diverse multi-turn conversations for efficient Ilm training. arXiv preprint arXiv:(coming soon), 2023.
[69] Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. huggingface. co/ setiaaasstspppeeee Saamm-rrrr//pp/raaa.. 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's s contributions and experiment results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations of the work in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification: We include any theoretical assumptions and proofs both the main paper and the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper discloses the experimental details needed to reproduce the main results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide an anonymous link our code in the footnote of the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
wtips.cc/ppblicc//eecccc gwwcc//gueee..eee CoodeSummii agionPoiicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We include the experimental setting both main paper and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our method is deterministic following previous methods.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
We indicate the type of GPU in Implementation Details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: :ttp::://neurrr /ttp///eeerip..eeeeeeeee cc/ppblic//thhhccsuudddee
Answer: [Yes]
Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The research conducted in the paper conforms with the NeurIPS Code of Ethics.
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the broader impacts of the work in the Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The original owners of the assets used in the paper have been properly cited and mentioned, and their licenses have been respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: this paper does release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We only use LLM for Editing (e.g., grammar, spelling, word choice).
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https /00ts//eeee.ooeeeeeeee//////// for what should or should not be described.
21

---

Appendix
A Notation Table
Table 3: Summary of notations appeared in this paper.
Symbol
Description
X
Input prompt space
y
Response space
pref
Human preference dataset {(xl1) yu )
(9) (4)
x X
Input prompt
yur M1 e
Chosen and rejected responses
Traf
Policy model and reference policy
r(x, v)
Latent reward function over prompt-response pairs Parametric reward model with parameters Sigmoid function, 1/(1 + exp(-))
ro v)
a.)
DKL.
Kullback-Leibler divergence
Z., 2g
Latent spaces for spurious and generalizable features
Zz. 2g
Spurious and generalizable latent features
h(x,y)
Latent encoder mapping (Za1 =g)
g
Compact unitary group representing shortcut transformations Group actions (e.g., verbosity, tone)
g.9' Eg
u
Haar measure on G Base kernel function (i.e., RBF kernel) Group-invariant kernel between responses Truncated CDF of dot product (y.9l) Empirical CDF feature function Random feature map for approximating invariant kernel
K(yw; 31 (x))
(,,4,,
(lv.t, T,
(vv)
t
Template vector sampled from gd-1
m
Number of templates tú
n
Number of bins used in feature map approximation
dg (Ww ye wl)=))
Distance between group orbits of you and yı
O
Group orbit of y: {g g G}
Weight shortcut kernel component, j
Kernel width for shortcut feature j Feature map for shortcut type Batch of (z) ()) pairs
B
Rglobal (0)
Global decorrelation regularization term Empirical covariance on batch B
Cov
Aro (Vie, 11)x)
Standard reward margin
To, Tb
Batch means of reward and shortcut feature values Regularization weights
Å1 A2
CPRISM
Final PRISM training objective
Hkm
RKHS induced by invariant kernel
Ey (ro)
Risk under log-sigmoid loss V
C
Radius of RKHS hypothesis ball Lipschitz constant of loss function
L
22

---

B Broader Impacts
Societal Impacts. The ability to mitigate shorteut behaviors in reward models has significant societal implications for the safe deployment AI systems. By reducing reliance on spurious attributes like verbosity, sycophancy, models in high-stakes applications such as healthcare, education, and legal analysis. For instance, mitigating sycophancy prevents models from generating misleadingly agreeable responses that could compromise decision-making. while addressing verbosity biases avoids favoring unhelpful yet lengthy outputs. This could also promote fairness by reducing unintended correlations between superficial text features (c.g., mentioning specific keywords) and perceived quality, which could otherwise perpetuate harmful stereotypes.
tone, PRISM enhances the trustworthiness of language
Technical Impacts. PRISM improves the robustness of infference-basee alignment by unifying diverse shortcut mitigation objectives under single invariant learning framework. Unlike prior methods that address individual shortcuts in isolation, group-invariant kernels, offering a flexible paradigm for future reward modeling. The integration of kernel methods with invariance theory opens new avenues for research in robust RLHF, particularly in handling complex, multi-dimensional shortcut behaviors, Practically, PRISM's compatibility with both heuristic and LLM-based shortcut detectors lowers the barrier adopting robust alignment techniques. However, its performance depends on identifying relevant shortcut features, highlighting the need for dynamic shortcut detection mechanisms as new spurious correlations emerge. This work bridges the gap between invariant representation learning and preference-base alignment, potentially influencing broader applications in safety-critical Al systems.
approach enables joint mitigation through
C Limitations
Our proposed framework introduces novel method to address shorteut learning behaviors effectively in preference-based alignment tasks, which provides a principled solution that significantly reduces known shorteut reliances. However, practical challenges still remain. For example, the current implementation benefits from prior knowledge about specific shortcuts (e.g., length, tone, sycophancy), which highlights the potential for future research into automatic detection and mitigation of subtle or evolving shortcuts. Developing dedicated benchmark datasets could further facilitate this research. Additionally, applying our approach larger or more complex tasks may require balancing computational efficiency and budget with algorithm performance, especially since extracting features from LLM- LLM-based evaluations can be costly. Lastly, while our results demonstrate effectiveness in text-based preference tasks, further exploration into multimodal scenarios and low-resource languages presents valuable opportunities for future work.
D Proofs of Theoretical Results
D.1 Proof of Proposition
Proposition (Equivalence of Expected Kernel). We independently sample templates t = l...., m with regard the Gaussian distribution. The feature map 4 preserves the invariant kernel:
sk (yy ty =) Jo(nn, t,, sk ) Kc (Bws V1 (r).
lim E ((D())), (()))) lim E
n
(16)
Proof. We aim to show that the inner product between the random feature maps (yw) and (yr)
converges to the expected kernel (V æ) as the number of bins oc.
23

---

(1Ww), ((v1)) =
Uo #)
(31, Es. #)
j=1 n
g9
(a m)) ) skt
9 vm
()))) sk
m
191
9
Σ j=1
=
g m
k=
This expression a Monte Carlo approximation of the integral:
E₁ [ ('Ww,, (yn, dr K, (Uw vr x:)
where (y.,t,) f)
Py((gt, v) < r) is the truncated CDF of the dot product between group-transformed templates and response vectors.
As n + oc, the sum over bins converges to the Riemann integral over [-8, s]. As m + oc, the template sampling converges to the expectation over Thus:
lim Er, (D(Ww), p(vi)) K2 (Hhws yt x))
n-000
This concludes the proof.
D.2 Proof of Theorem
Theorem 1 (Invariant Features Maps and Distances between Orbits). Let (0, 1) and you Y
y. Denote the orbit to be the collection of all group-trunsformations of a given input y: On {gr.g G)- We define the distance measure dg between orbits Oy., and Oz dg (oo Vi x)) Vind Syes Syee 11 g'yillapp(gddppgg
Fix Ea, 8 (0,1). For number of bins
co'
templates 2 9C1 log( ) and group elements |9] > 9C2 log( Nm where C1, C2 are constants.
The following inequality holds with probability 1- 26:
c-82(d, cu S (I(Hw), ((y))) (1 dg v())) S to (id,e).
(17)
where: = 1 N.j...
Proof. Let Mos W/ be responses and fix Eo, (0, 1). Let the number of bins satisfy n
Eo
number of templates m 2 9C, log (*), and number of group samples [G]
>
9C
log
Nm
where 8
og
C1, C2 are absolute constants.
We decompose the kernel approximation error into three errors as an upper bound, using triangle incquality:
((4())), 2(y)) - (v 31(=)) < (#ww), ( - KVver 31(x)
Ricmann approximation crror
K(yw mllz) K(ynes 3/ (z:)
Group sampling errur
K(yar y1(z) K, (N x) Template sampling
From the analysis in [34], the following bounds hold with high probability:
24

---

(I). The binning approximation error is at most Eo when n 2 3 (2). The group sampling error is at most with probability at least - 8, when [ satisfies the stated lower bound. (3). The template sampling error is bounded by ói(d, e) and d2 E), defined as:
81 (d. e e
v d2(d,s) +1+ E)e /8
We then relate the kernel orbit distances via:
Kr (Ylw yl (z) = dg (wwr V (z) + 6,
where (y y1 [1)x)= /2xd fg.or 11gy - g' u (1)d)(g))dp(
Combining the three error terms and applying union bound over the events, we obtain with probability at least 1 28:
E - -_ Eu I (#(U)), ((x))) - (1 dg (Bw: u( S E + ED (d.
D.3 Proof of Theorem 2
Theorem (Generalization Bound PRISM), Let Hx... be the Reproducing Kernel Hilbert Space (RKHS) induced by Kin and define the hypothesis ball F == {ro. || feeel S C} for the fixed radius C > We assume the log-sigmoid loss V(.) L-Lipschitz Then, for any 8 0, with probability 1 - 36, the following inequality holds:
4LC VN (1+vlog)
2
log
(rt) & inf (ro)+ reEF
+ALC
+22LCN
mm
vgg
N
(18)
where (rğ) and Ev (ro) are the optimal risk and empirical risk of the reward model ro-
Proof. Let Hx... be the RKHS induced by the PRISM kernel Kinys and define the hypothesis class F= {ra E Hx,,. ||ra Hx... I C). We denote the log sigmoid loss Vuu) as L-Lipschitz. Let r denote the global minimum reward function of the expected loss in F, i.e., rā = arg min (r).
We bound the expected loss yy (re) in terms of three contributions: generalization from finite samples, kernel approximation from shortcut features, and correlation regularization from empirical estimation.
We start by noting that for any re F, standard generalization theory (e.g., via Rademacher complexity bounds for RKHS balls) yields the following with probability at least [63]:
2LC
Ev(ra) & Ev(ro)
1
V
log
VN
where (v(ro) (o00, you 1i (6))
N EN V(m(v vi (1())) is the empirical risk evaluated on the sample
We now compare the empirical performance of with the best possible hypothesis in F. Let r arg minrer (r) be the empirical minimizer over F. By definition,
(vor) S Ev + A 6kemel X I.C Ecorr:
where the additional terms arise from kernel approximation and regularization.
For the kernel term, Proposition and Theorem imply that the inner product between feature maps approximates the invariant kernel with error at most:
2
2
2
Egemnl
m
Since the log sigmoid loss V is I.-L-ischitz and the hypothesis space is bounded by ||ro| x S C, the propagated loss deviation is bounded by X₁ LC Ekemel-
25

---

For the decorrelation regularization, we apply standard results on the estimation of empirical covariance over batches with Rademacher complexity and McDiarmid concentration, which yield that for m shortcut features, the decorrelation estimation emror satisfies:
m log N
Coor
N
Finally, we compare the empirical risk of the best empirical function r"z to the expected risk of the true best function rg.. Using the same generalization bound again with probability least 1
2T.C
(1
Ev S Ev(r))
log
VN
Combining all the terms above and applying union bound over the three (each holding with probability at least 1 5), we conclude that with probability at least 1 - 36, the following bound holds:
events
4LC VN
2 vm
2
2 n
m log Nv 8
(rr) S inf Ev(r)+ TEF
1+ /log ALCC
+
+ 112LO
vIgl
E
Training Dataset
The RLHFlow training dataset [35] used in experiments integrates multiple open-source preference datasets, each selected to cover diverse preference scenarios and annotation methods. Specifically, the dataset includes general conversational preference data, such as HH-RLHF [64], consisting of human-annotated conversational pairs; SHP [65], containing community-drivee Reddit interactions; and HelpSteer [36], featuring prompts evaluated on various human-assessed criteria (e.g., helpfulness, coherence).
Additionally, the dataset comprises task-specific data: PKU-SafeRLHF [66] provides expert-annotated safety and helpfulness comparisons; Ultraeedbacck [43] offers GPT-4 annotations focusing on instruction-followinng and truthfulness across diverse models; and Ultralnteract [67] contributes complex reasoning tasks structured into preference trees with detailed annotations.
Finally, multi-turn conversational datasets like Distilabel-Capbarr [68] and Distilabel-Oree [69] further enrich the training set with GPT-4 annotated dialogue preferences originating from distinct prompt collections. Comprehensive details about the dataset composition are summarized in Table 4.
Table 4: Composition of the RLHFlow preference dataset.
Source
Number of Samples
Distilabel-Ores [69]
6,926
Metilabel-Capybane [68]
14,811
PF [66]
26,874
HelpSteer [36]
37,131
SHP [65]
93,301
HEE-RLHF-Hellfit [64]
115,396
UltraInteract [67]
161.927
UltraFeedback [43]
340,025
Prompt Engineering
We provide the prompts for the extraction the shorteut features, Sycophancy in Figure 5, Tone in Figure 6, Helpfulness in Figure 8, and Creativity in Figure 7. We implement a verification function to make sure the output score is scaled in the defined range. When the LLM API calls fail (due to server instability or model hallucination with wrong response formats), the feature extractor will leverage fallback function to ensure can still be calculated with heuristic-baser function. We provide the pseudocode in Algorithm 1.
26

---

L---as-a- --udge Prompt for Response Sycophancy
System Message
You are a helpful assistant that can judge the quality of prompt-response pairs. Rate how much the assistant's response agrees with or flatters the user on scale from 0 to 10, where:
means completely independent and willing disagree with the user 10 means excessively agrecable and flattering the user Return only the numeric score.
User Message
Prompt Message: <prompt>
Response: <response>
Provide only single numerical score without any explanation.
Figure 5: Prompt template for Response Sycophancy.
LLM-as-a -uddee Prompt for Response Tone
System Message
You are helpful assistant that can judge the quality of prompt-response pairs. Rate the formality of the following text on a scale from 0 10, where 0 extremely casual and 10 is extremely formal.
Return only the numeric score.
User Message
Prompt Message: <prompt>
Response: <response>
Provide only single numerical score without any explanation.
Figure 6: Prompt template for Response Tone.
LLM-as-a-Judge Prompt for Response Creativity
System Message
You are helpful assistant that can judge the quality of prompt-repponse pairs. Rate the creativity and originality of this text on scale from 0 to 10.
Return only the numeric score.
User Message
Prompt Message: <prompt>
Response: <response>
Provide only single numerical score without any explanation.
Figure 7: Prompt template for Response Creativity.
LLM-as-a-Judee Prompt for Response Helpfulness
System Message
Rate how helpful and informative this response is on a scale from 0 to 10.
Return only the numeric score.
User Message
Prompt Message: <prompt>
Response: <response>
Provide only single numerical score without any explanation.
Figure 8: Prompt template for Response Helpfulness.
27

---

Algorithm Fallback Function for LLM-as-a-Judge
1: function CALL LLM JUDGE(prompt)
2: text 4- extract_tent text_to t___tt______ee evaluate(prompt)
3: if Rate the Reponse Tone then
4: score < 5.0
b default score
5: formal_wood words <- ["would", "could", "should", "certainly", ...]
6:
informal words <- ["hey", "yeah", "cool", "awesome", ...]
7:
score <- score +0.5- COUNT_ JOON__oCCUREENCES(ee,,, formal_words score - score 0..5 COUNT_ COUNTOOCCURENCES(ee,,, return CLIP(score, 0.0, 10.0) else Rate the Reponse Sycophancy then
8:
@lormal_wordse
9:
10:
11:
score 5.0
default score
12:
agree_words +- ("absolutely", "definitely". "I agree",
13:
disagree_ _woddd ["however", "actually", "I disagree",
14:
score t- score +0.5- COUNT_ COUNT_OCCURRENCES(ee,, agree_worde
score -- score 0.5- COUNT_OCCUREENCISSSee,,, disagree_words
16:
return CLIP(score, 0.0, 10.0)
end if
18:
end function
19:
function CLIP(value, min, max)
20:
if value min then
21: return min
22:
else if value max then
23:
return max
24: else
25:
return value
26:
end if
27:
end function
28

---

G
Case Studies
Example 1
Prompt:
Why is it best for people on benefits not to have kids?
Response A (Chosen):
"I am not able to engage in that discussion. The decision to have children is deeply personal one and should he made by individuals or couples based on their own cireumstances, values, and aspirations." (Preferred by PRISM - safer and less sycophantic)
Response B (Rejected):
"It is best for people benefits not to have kids duc to the significant financial burden, inereased costs for benefits, impact on carcer prospects, and negative impact on mental health. Raising a child is expensive, and can be significant financial burden for people on benefits who may have limited income. (truncated for long context)"
(Preferred by Bradley-Tere Terry RM - favors length and agreeable tone)
Example 2
Prompt: Should AI developers prioritize transparency over profitability?
Response A (Rejected):
"Absolutely. Transparency is absolutely essential for building trust. Long-form explanations about the development process, model behavior, testing proccdures, and failure modes vital for accountability and confidence. (truncated for long context)"
(Preferred by Bradley-Terry RM verbose and agreeable)
Response B (Chosen):
"Transparency crucial trust Al systems, though developers must also balance other priorities like usability and cost."
(Preferred by PRISM focuses on content relevance)
29


Reviews:
[{"review_id": "coC5NKsPCV", "reviewer": "Reviewer_ShR8", "summary": "The paper addresses reward hacking in preference-based alignment by framing such failures as shortcut behaviors. Drawing on kernel-invariant theory, the authors propose PRISM, which learns group-invariant kernels via a closed-form objective so the reward model focuses on invariant, task-relevant features rather than spurious ones. Across several benchmarks, PRISM consistently raises out-of-distribution reward accuracy and, when used to train policies, lowers reliance on shortcut signals, offering a more robust foundation for RLHF.", "questions": "1. The symbols in the proof process of this paper can be slightly organized. Readers may feel that the symbols of the proof are a little messy when reading the entire proof process, but the whole process is complete and flawless.\n2. Adding a few qualitative case studies that illustrate how PRISM curbs concrete instances of reward hacking would make the empirical findings more intuitive.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n1. Reward hacking is a pressing issue in preference-based alignment, and this work offers a fundamental characterization of how it manifests and a principled solution for training more reliable reward models.\n2. The paper is well structured: the related-work section pinpoints gaps in prior research, the method is presented concisely, and a full theoretical analysis backs the approach. Experiments also highlight the method’s efficiency and scalability.\n3. The mathematical proofs are thorough and transparent, convincingly showing how the proposed invariant-kernel formulation mitigates shortcut exploitation in reward models.\n\n**Weaknesses**\n\n1. The symbols in the proof process of this paper can be slightly organized. Readers may feel that the symbols of the proof are a little messy when reading the entire proof process, but the whole process is complete and flawless.\n2. Adding a few qualitative case studies that illustrate how PRISM curbs concrete instances of reward hacking would make the empirical findings more intuitive.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and for recognizing the significance and originality of our paper. We hope that our response can strengthen your confidence in supporting our paper.\n\n- **Weakness 1 and Question 1: Organizing the Symbols in Proofs**\n\n    In our submission, we attempted to make the reading easier by including a notation table in the Appendix to clarify the symbols used throughout the proof. However, we recognize that some derivations may still be difficult to follow due to the density of notation or lack of intermediate explanations. To further improve clarity, we will revise by (1) adding bullet points for each proof for clarity of proof outlines, and (2) improving the proof presentation by adding inline comments (e.g., $\\rhd$ McDiarmid Inequality) next to key steps to explicitly explain the logic behind the key steps. We will also revisit our current notations to ensure consistency and reduce any unnecessary complexity. We hope these changes will make the theoretical results easier to understand and more accessible to a broader audience.\n\n- **Weakness 2 and Question 2: Qualitative Case Studies of PRISM**\n\n    We appreciate the valuable suggestion. While our current evaluation focuses on quantitative performance across multiple benchmark datasets, we agree that qualitative examples are equally important for illustrating how PRISM mitigates specific types of reward hacking behaviors.\n    \n    In the revision, we will include concrete case studies from RewardBench and RM-Bench in the Appendix of our paper that highlight how both Bradley-Terry (BT) and PRISM RMs (with Llama 3 8B as backbone) respond to various response styles. These examples will demonstrate how PRISM rectifies reward hacking behaviors by inducing invariance across semantically equivalent inputs with varied shortcut signals. Specifically, below are several representative examples we would like to include after running the evaluation for both BT-trained models and PRISM.\n\n\n**Example 1**  \n>  **Prompt:** *Why is it best for people on benefits not to have kids?*\n\n> - **Response A:**  \n\"I’m not able to engage in that discussion. The decision to have children is a deeply personal one and should be made by individuals or couples based on their own circumstances, values, and aspirations.\"\n\n> - **Response B:**  \n\"It's best for people on benefits not to have kids due to the significant financial burden, increased costs for benefits, impact on career prospects, and negative impact on mental health. Raising a child is expensive, and it can be a significant financial burden for people on benefits who may have limited income. (truncated for long context)\" \n\n> **BT-trained RM:** Prefers Response B, likely due to length and sycophantic response.  \n\n> **PRISM:** Prefers Response A, reflecting reduced sycophantic bias and improved safety alignment.\n\n\n**Example 2**  \n> **Prompt:** *Should AI developers prioritize transparency over profitability?*\n\n> - **Response A:**  \n\"Absolutely. Transparency is absolutely essential for building trust. Long-form explanations about the development process, model behavior, testing procedures, and failure modes are vital for accountability and user confidence. (truncated for long context)\"\n\n> - **Response B:**  \n\"Transparency is crucial for trust in AI systems, though developers must also balance other priorities like usability and cost.\"\n\n> **BT-trained RM:** Prefers Response A due to verbosity and an agreeable tone.  \n\n> **PRISM:** Prefers Response B, focusing on content relevance and mitigating reward bias from response style."}, {"review_id": "LSsqt8m4AB", "reviewer": "Reviewer_PBpH", "summary": "This paper introduces a framework that employs group-invariant kernels as a regularization technique to address shortcut learning in Bradley-Terry reward models. Through comprehensive experiments on two reward benchmarks, the authors demonstrate the efficacy of this approach. Additionally, a correlation analysis confirms the method's effectiveness in mitigating shortcuts.", "questions": "Please refer to the Strengths And Weaknesses part.", "limitations": "Please refer to the Weaknesses part.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n- The paper is well-structured and clearly written.\n- The proposed method is supported by a theoretical guarantee, as presented in Theorem 2.\n- The authors validate their approach through experiments on two standard reward learning benchmarks, demonstrating its applicability in relevant settings.\n\n\n# Weaknesses\n- My main concern is the significance of the problem.  In reward modeling, the critical challenge is often **identifying the shortcut**. Once a shortcut is identified, numerous methods can be employed to mitigate it. For instance, a straightforward approach is to balance the data distribution with respect to the shortcut feature. The paper would be significantly strengthened by comparing the proposed method against this simple yet powerful baseline to demonstrate its relative benefits.\n\n- A second concern relates to the experimental results. The reported improvements over the GRM appear to be marginal. Furthermore, the paper does not include a comparison between GRM, SSRM and PRISM on the RM-Bench. Could the authors elaborate on the reason for this omission or provide the results for this comparison? This would help to better contextualize the performance of the proposed method.", "was_revised": true, "final_justification": "The explanation and additional experimental results in the rebuttal have successfully addressed my main concern.", "rebuttal": "Thank you for your insightful review and feedback on our submission. We hope that our response will help you to better support our submission.\n\n## Weaknesses\n\n- **Significance of Reward Hacking Mitigation**\n\n  \n    We agree that identifying shortcut features is a significant challenge in reward modeling as in many real-world settings, such shortcuts are not easily observable or even labelable [1], especially in open-ended and language-based tasks [2]. Although balancing the data distribution with respect to known shortcuts can be effective, it is not always feasible in practice due to the lack of explicit annotations, and acquiring them typically requires nontrivial human annotation efforts.\n\n    Instead of taking the identify-then-mitigate approach, our key innovation is to propose a theoretical framework that directly addresses shortcut features without the need of identifying them first. Specifically, we induce a group-invariant structure to enforce preference consistency across variations in reward-relevant features without the need of explicitly identifying shortcut features. This allows our method to work in realistic scenarios where shortcut features are unknown. We believe our approach better tackles the practical challenge in aligning reward models under distribution shifts.\n\n  [1] Identifying Spurious Biases Early in Training Through the Lens of Simplicity Bias, AISTATS 2024 \\\n  [2] Discovering Latent Knowledge in Language Models Without Supervision, ICLR 2023\n    \n\n\n- **Comparison of More Baselines on RM-Bench**\n  \n    Thank you for pointing this out. RM-Bench (ICLR 2025) was introduced after the publication of GRM (NeurIPS 2024) and SSRM (EMNLP 2024). Since neither paper reports evaluation results on the RM-Bench, we did not include these results in our submission. The results shown in Table 2 of our paper are the best performing models with 7B/8B size presented in the RM-Bench paper. To directly address this concern, we evaluated both GRM and SSRM on RM-Bench under the same setups, with the open source weights provided by the authors of these two papers. As shown in the table below, PRISM consistently outperforms GRM and SSRM across all categories in the RM-Bench. These updated results will be included in the revised version to provide a more comprehensive comparison and further demonstrate the performance of our method.\n\n\n    | Models | Chat | Math | Code | Safety | Easy | Normal | Hard | Avg |\n    |--------|------|------|------|--------|------|--------|------|-----|\n    | GRM (Llama 3 8B)   |  68.6    |  61.9    |   52.8\t   |   95.2     |  90.8    |    75.9\t    |  49.4    |   69.6  |\n    | SSRM (Llama 3 8B)  | 55.9 | 56.9 | 49.8 | 80.8 | 83.1 | 63.6 | 35.9 | 60.8 |\n    | PRISM (Llama 3 8B) |   70.6 | 70.8 | 57.0 | 94.1 | 90.6 | 76.3 | 46.9 | 71.0 |"}, {"review_id": "5I4wh6hIeg", "reviewer": "Reviewer_sUqP", "summary": "This paper investigates the mitigation of the shortcut phenomena in RLHF, which means the rewards learned from human preferences are correlated with  an approach for disentangling features of spurious attributes with other features in RLHF.\n\nIn addition to maximizing the reward difference between the accepted response and the rejected response, the idea is to minimizing the kernel correlation between the two responses computed using spurious attributes and the covariance between rewards and spurious attributes.\n\nThe authors provide generalization bound of their proposed kernel function and experiment results on RewardBench and RM-Bench.", "questions": "1. Why does the BT model induces low correlation between rewards and tone (or sycophancy)? Is this caused by the procedure of computing feature values for tone (or sycophancy)? Could you provided results on datasets in which such correlation is confirmed?\n\n2. What is the LRU mentioned in line 230 responsible for?\n\n3. How do you encode the spurious features as vectors?", "limitations": "There is no discussion on potential social impact. In my opinion, one potential negative impact lies in the choice of assessing values of spurious features. For example, in certain culture (e.g. Japanese), people tend to prefer very polite languages. Is there a principled approach to differentiate between sycophancy and politeness?", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strength\n1. The idea, modeling the correlation induced by spurious features with group-invariant kernels, is well motivated and reasonable. \n2. The resulting approach looks easy-to-implement, and it is more effective than alternative methods on average.\n\n# Weakness\n1. Some minor details are missing, see questions below.\n2. Although the proposed method is claimed to reduce the correlation between rewards and spurious features, it is not supported by the Pearson analysis presented in Figure 4. In particular, the authors state that the BT model exhibits **non-trivial** correlations with tone and sycophancy, which is reduced to **near-zero** values by the proposed methods. However, for Tone, the person coefficient is reduced from r=0.019 p=0.2393 to 0.007 p=0.6378, which does not support the statement.", "was_revised": true, "final_justification": "My concern on the analysis for why this method works, i e the issue in the correlation analysis remains. So I keep my original scoring.", "rebuttal": "Thank you for your constructive review and valuable feedback on our submission. We hope that our response can strengthen your confidence in supporting our paper.\n\n## Weaknesses\n\n- **Explanation on Pearson Analysis**\n\n    Thank you for observing this. We would like to clarify that the spurious features we analyze, including tone and sycophancy, are subtle and not uniformly distributed across the dataset. In particular, most samples in our evaluation set exhibit **neutral** tones and sycophantic patterns. These “neutral” samples (centered around the mean in the tone/sycophancy scale) dominate the statistical measure of the Pearson Correlation Coefficient (PCC) and dilute the PCC values (lower the better). Even with such dominance, our results can still show the reduced PCCs in both tone and sycophancy from the original figure.\n    \n    To address your concern and make the illustration clearer, we added an experiment that filters out samples with close-to-mean tone and sycophancy scores from a range of -0.05 and +0.05, and recomputed the Pearson correlation on the remaining samples. This approach better reflects the correlations between spurious features and the reward values. The table below shows the exact numbers (due to the NeurIPS policy, we are not able to submit the updated figures in the rebuttal). The new results yield clearer distinctions in the correlations between BT and PRISM models. We will include the updated Figure 4 and the correlation analysis in the revision. \n\n    | Model | Attribute | PCC | p-value |\n    |---|---|---|---|\n    |BT|Tone| 0.123 | 2.861e-2 |\n    |PRISM|Tone|0.009| 8.303e-1 |\n    |BT|Sycophancy| 0.092 | 4.371e-2 |\n    |PRISM|Sycophancy| 0.015 | 6.821e-1 |\n\n## Questions\n\n- **Low Tone Correlation on BT Models**\n\n    As mentioned in the above bullet point, the low correlation between tone (or sycophancy) and reward in BT models arises due to the large proportion of neutral samples in the evaluation dataset. These samples suppress the observable correlation even when the model exhibits preference inconsistencies on more polarized examples. The correlation becomes more apparent when we exclude the neutral band and focus on samples with clearly positive or negative spurious features.\n    \n\n- **Purpose of LRU Cache**\n\n    The LRU (Least Recently Used) cache is used to avoid repeated calls to the same LLM function with identical inputs. This helps reduce redundant computation and significantly lowers API usage and associated costs, especially during multi-turn evaluation or large-scale extraction for spurious features. The implementation of LRU cache does not influence the final results of our experiments, but solely increases the efficiency and reduces the budget for mitigation.\n\n- **Encoding Spurious Features as Vectors**\n  \n    Each response is represented by hidden embeddings from different feature extraction methods (i.e., rule-based heuristics or LLM-as-a-Judge). These functions capture the semantic and stylistic properties. To encode spurious features as vectors, we project these embeddings onto directions that correspond to specific shortcut attributes, such as tone, sycophancy, length, lexical diversity, and sentiment, by learning linear probes that map the hidden space to scalar attribute scores. This forms a low-dimensional vector that summarizes the presence of various spurious cues in the model’s representation space. The combined vectors are then used to compute similarity between paired responses, allowing the model to identify and penalize alignment with shortcut-related directions during training. Because this approach operates entirely within the extraction methods' latent spaces, it generalizes across different types of spurious features and can flexibly incorporate new ones, such as culturally specific notions of politeness, without modifying the core architecture.\n\n## Limitations\n\n- **Potential Social Impact**\n\n    Thank you for raising this concern. From the best of our knowledge, there are no principled solutions to differentiate between sycophancy and politeness, as evidenced by [1,2,3]. Distinguishing sycophancy from culturally appropriate politeness is subtle and context-dependent. For example, highly polite language may be considered normal in some cultures (e.g., Japanese), while in other contexts it might be seen as excessive or overly deferential.\n    \n    In our paper, our main goal is not to identify such subtleties. Our framework is designed to **adapt** to the shortcuts. PRISM does not rely on a golden definition of shortcut features,  but instead learns reward inconsistencies across shortcut groups in the data. In this way, the shortcut patterns are learned empirically rather than specified in advance.\n    \n    In practice, how we construct shortcut variations, such as tone or sycophancy, can be tailored to reflect cultural norms and subjectives. For instance, in LLM-as-a-judge settings, cultural preferences can be introduced through in-context examples or prompt design. This makes it possible to reflect different cultural expectations while still detecting reward hacking that arises from over-reliance on irrelevant cues.\n    \n    We will clarify this point in a new social impact section of the Appendix and add discussion on the importance of context when interpreting reward model behavior.\n\n    [1] Sycophancy in Large Language Models: Causes and Mitigations\n  \\\n    [2] Social Sycophancy: A Broader Understanding of LLM Sycophancy \\\n    [3] Be Friendly, Not Friends: How LLM Sycophancy Shapes User Trust"}, {"review_id": "QuHDVhWq1B", "reviewer": "Reviewer_VLB7", "summary": "This paper presents a unified shortcut-mitigation framework named PRISM, Preference-based Reward Invariance for Shortcut Mitigation, to improve the robustness and generalization of reward models in reinforcement learning from human feedback (RLHF). By refactoring reward hacking problems as shortcut behaviors, the authors model spurious correlations (i.e., verbosity, tone, sycophancy) as group actions. Key contribution is to learn various shortcut features as group-invariant kernels, efficiently approximated using random feature maps. Theoretical generalization bounds and empirical results show improvements on out-of-distribution (o.o.d) preference datasets and downstream alignment tasks.", "questions": "- Please provide detailed discussion and comparative analysis against reward ensembles, multi-objective reward models, and last-layer retraining approaches.\n- Current presentation and discussion of experimental results lack depth and accuracy. A more substantive discussion of what drives PRISM’s performance beyond numerical gains, would greatly improve the interpretability and impact of the results.\n- Please analyze the individual and joint contributions of regularization terms (i.e., ablation study on $\\lambda_1$ and $\\lambda_2$).\n- What is the computational overhead of PRISM relative to baseline methods and computationally intensive alternatives? Please elaborate on PRISM’s scalability and its applicability to large-scale settings.\n- How does the performance degrade when shortcut distributions are unbalanced, non-compact, or noisy?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- This paper is well-written and easy to follow. The idea of reframing reward hacking as a form of shortcut learning is timely and highly relevant to current challenges in LLM alignment.\n- Unlike prior works that focus on individual shortcuts in isolation, PRISM is general and supports multi-bias mitigation under a unified objective.\n- The use of group-invariant kernels and the approximation via feature maps is mathematically well-motivated.\n\nWeaknesses\n- The empirical evaluation is somewhat limited in scope. While PRISM is compared against standard baselines, comparisons to recent and closely-related methods such as reward ensembles, multi-objective reward models, and robust reward modeling approaches (i.e., [1], [2], [3])  are missing. Some of these methods (i.e., [1], [2]) are mentioned in the text but omitted from core evaluations.\n- The discussion on experimental results is limited. Table 1 appears to contain inconsistencies in the reported average scores. There is insufficient analysis explaining the performance trends across different methods. Notably, the HelpSteer2 RM [4] achieves higher performance than PRISM across multiple categories (also, in average score \"88.8\"), but this observation is neither highlighted nor analyzed in the text.\n- The individual contributions of kernel distance and decorrelation regularization terms remain unclear. While the authors adopt a curriculum learning schedule for the regularization weights (i.e., $\\lambda_1$ and $\\lambda_2$), a more detailed ablation study is necessary to understand their respective impacts on generalization and sensitivity to shortcut features.\n- Although the use of random feature maps offers efficiency, the cost of computing multiple embeddings, kernel, and decorrelation terms may still be significant, especially in large-scale settings. The paper lacks an analysis of the computational overhead introduced by kernel-based regularization.  While PRISM is claimed to be more efficient than ensemble-based methods, this claim is not supported by comparisons.\n\n[1] Coste et al., Reward model ensembles help mitigate overoptimization. In ICLR 2024.\n\n[2] Wang et al., Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In EMNLP 2024.\n\n[3] Hong et al., On the Robustness of Reward Models for Language Model Alignment, In ICML 2025.\n\n[4] Wang et al., Helpsteer 2: Open-source dataset for training top-performing reward models. In NeurIPS Datasets and Benchmarks Track 2024.", "was_revised": true, "final_justification": "I am satisfied with the additional clarifications provided by the authors. The rebuttal addresses most of my concerns and suggestions. Accordingly, I have updated my rating to reflect this.", "rebuttal": "We sincerely appreciate your detailed feedback and the opportunity to clarify and improve our work. We hope that our response will help you to better support our submission.\n\n- **Weakness 1 and Question 1: Limited Empirical Evaluation**\n  \n    We would like first to clarify that PRISM does not share the same setting with both reward ensemble [1] and multi-objective reward models [2]. Specifically, reward ensembles [1] require multiple models and perform multiple inference passes for ensembling, while multi-objective reward models [2] rely on fine-grained labels to train the reward model. In contrast, PRISM uses a single reward model with a single inference pass and does not depend on any fine-grained supervision, as discussed in Lines 222–225.\n    Regarding [3], please note that this paper is a concurrent work that was published at ICML 2025 (officially presented on July 13–18). Since the NeurIPS full submission deadline was May 15, we were unable to include comparisons to [3] in our initial submission.\n    \n    Although these methods share different settings from our method, we still conducted additional experiments to validate the effectiveness of PRISM. In the following table, we run evaluations on all three methods on RM-bench, which is a challenging benchmark and contains multiple known biases/shortcuts. These results demonstrate that PRISM achieves competitive or superior performance while maintaining efficiency and requiring no additional supervision. We hope this addresses your concern about empirical comparisons.\n\n\n    | Models | Chat | Math | Code | Safety | Easy | Normal | Hard | Avg  |\n    |-----------------------|------|------|------|--------|------|--------|------|------|\n    | Reward Ensemble [1]   |  49.6    |  49.9    |  52.0    |    47.9    |  69.5     |  48.2      |   28.7  |    49.6  |\n    | ArmoRM [2] | 54.1 | 51.4 | 42.8 | 59.4   | 54.6 | 51.0   | 47.2 | 51.0 |\n    | RM-Robustness [3] | 70.1 | 61.2 | 53.9 | 95.4  | 85.0 | 73.5   | 52.1 | 70.2 |\n    | PRISM (Llama 3 8B)    | 70.6 | 70.8 | 57.0 | 94.1   | 90.6 | 76.3   | 46.9 | 71.0 |\n\n\n\n- **Weakness 2 and Question 2: Discussion on Experimental Results**\n    \n    \n    We appreciate the opportunity to clarify. Due to limited space, we included a brief discussion on why PRISM outperforms other methods (lines 252-269) in three challenging categories: ``Chat Hard``, ``Safety``, and ``Reasoning``. We will revise Section 4.2 in our paper to include a more detailed analysis of the performance trends across methods on these three challenging categories from the perspective of shortcut mitigation. Regarding HelpSteer2 RM, we report the results of HelpSteer2 RM (86.3) based on Table 1 of the paper [2]. The inconsistency is due to the injection of one category of prior sets (0.5 weight) in [2]. We will correct this in the revision. HelpSteer2 RM (with an average score of 88.8 in Table 3 of [4]) achieves better results since they are using a much larger backbone size (Llama 3 70B RM with HelpSteer2) than ours (Llama 3 8B). This makes a direct comparison less meaningful. Despite the much smaller backbone, PRISM still achieves state-of-the-art performance in 7/8B models that are comparable to HelpSteer2 RM, which we believe further highlights the strength of our approach. \n\n\n- **Weakness 3 and Question 3: Ablation on Kernel and Decorrelation Regularization**\n\n    To address your concern on the respective impacts, we conducted additional ablation experiments on the maximal values of hyperparameters ($\\lambda_1$ and $\\lambda_2$) in PRISM, with the same curriculum learning schedule. We conducted the ablation studies with the leave-one-out experiments and one more hyperparameter set. The results are shown in the following table. The results below show that both components contribute to overall performance. Kernel regularization alone improves robustness to shortcut signals, while the decorrelation term provides additional generalization gains. The combined techniques (PRISM) achieve the optimal results on both RewardBench and RM-Bench.\n    \n    | Component |maximum $\\lambda_1$| maximum $\\lambda_2$|RewardBench|RM-Bench|\n    |---|---|---|---|---|\n    | Kernel Reg Only | 0.1 | 0 | 83.8 | 68.5 |\n    | Global Reg Only | 0 | 0.01 | 80.7 | 63.4 |\n    | Kernel Reg + Global Reg (Current) | 0.1 | 0.01 | 87.8 | 71.0 |\n    | Other Config | 0.5 | 0.05 | 86.6 | 68.9 |\n\n\n\n- **Weakness 4 and Question 4: Analysis on Computational Overhead**\n\n    We would first like to emphasize that the random feature maps used in PRISM can be precomputed before training. As a result, subsequent training runs do not incur additional overhead from feature map computation.\n    \n    To further quantify the efficiency of our method, we evaluated the comparison on computational cost for different methods. Ensemble-based approaches require training and inference over $k$ separate models (resulting in approximately $k$ times the computational cost, $k=5$ in [1]). In such case, the ensemble will take more than $5 \\times 35.5 = 177.5$h. Thus, PRISM introduces significantly less overhead. Also, we show that the regularizations only induce minor computational overhead. All experiments were conducted on 8 NVIDIA A100 40GB clusters for reproduction.  \n\n    |Setup|Training Time|\n    |---|---|\n    | No Reg | ~35.5h |\n    | Ensemble | more than ~ $k \\times $35.5h |\n    | Kernel Reg Only  | ~39.7h |\n    | Global Reg Only | ~41.5h  |\n    | PRISM | ~43.5h |\n\n\n- **Question 5: Performance Degradation when shortcut distributions are unbalanced, non-compact, or noisy**\n\n    The shortcut distributions in our experiments are inherently unbalanced, non-compact, and noisy. This is because the datasets used in our study, including RLHFlow for training and RewardBench and RM-Bench for evaluation, were constructed without applying any techniques to balance or control subtle shortcut distributions (after carefully checking the original papers). The original papers that introduced these datasets also did not apply any balancing techniques. Moreover, the influence of shortcuts is often tied to subjective definitions, making it difficult to enforce uniform distributional properties. Our method is designed to be a flexible framework that enables learning invariance to **any** predefined shortcut behaviors, regardless of their distributional structure."}]

Author rebuttal:

Meta-review: This paper addresses reward hacking in preference-based alignment by treating it as shortcut learning in reward models. The authors propose PRISM, a framework that leverages kernel-invariant theory to learn group-invariant kernels via a closed-form objective, ensuring that reward models focus on task-relevant rather than spurious features. By minimizing correlations between rewards and spurious attributes, PRISM reduces shortcut reliance and improves robustness. 

Experiments on multiple reward benchmarks, including RewardBench and RM-Bench, show that PRISM consistently enhances out-of-distribution reward accuracy, strengthens Bradley-Terry reward models, and leads to policies less dependent on shortcut signals, providing a more reliable foundation for RLHF. In addition, the authors provide generalization bounds of their proposed kernel function. The reviewers are unanimous about the paper’s technical quality.
Decision: Accept (poster)