A Descriptive and Normative Theory of Human Beliefs in RLHF

## Abstract

Abstract Human preferences in RLHF are typically modeled as a function of the human’s reward function or corresponding optimal state-action values. In this work, we propose that human beliefs about the capabilities of the agent being trained also play a key role in preference generation. We examine two questions related to this hypothesis, one descriptive and one normative, respectively: Do human labelers’ beliefs about agent capabilities affect the preferences that they provide? And what is the ideal set of beliefs about an agent—and resulting preferences—for humans to have? We propose a new preference model that incorporates human beliefs and provide a normative theory that bounds the error on the final learned policy based on the mismatch between the human’s beliefs and an idealized set of beliefs. We then confirm via a human study that beliefs about agent capabilities do, in fact, significantly affect preferences and can be influenced through simple interventions. Additionally, we empirically show through synthetic experiments that it is often suboptimal for human preference labelers to assume agent optimality. Collectively, these results theoretically and empirically demonstrate how reducing the mismatch between human beliefs and agent capabilities can lead to more performant RLHF and point toward new best practices for RLHF practitioners.

## 1 Introduction

Reinforcement learning from human feedback (RLHF) is one of the main tools used to align powerful AI systems (Kaufmann et al., 2023). Alignment is important for numerous reasons, such as minimizing unintentional harm, ensuring safety and control, and increasing public trust and legal compliance (Ji et al., 2023; Amodei et al., 2016). In order to use RLHF for alignment more effectively, researchers require high-quality data from humans that express rational preferences. This begs the question: How should rationality of preferences be defined within RLHF? Standard approaches to RLHF assume that humans provide preferences based only on what actually happened in the trajectories, i.e., the return-based interpretation (Christiano et al., 2017; Ziegler et al., 2019; Ouyang et al., 2022; Brown et al., 2019). This partial return-based approach does not consider events that could have occurred while taking a risky action or what might happen after the end of the trajectory segment. Knox et al. show that the regret-based model addresses this by looking at the quality of actions taken, rather than just the summed reward of the observed outcomes (2024).

Figure: Figure 1: Illustration of scenario in which preference for a more optimal partial path can lead to a worse post-RLHF policy. (Path 1): The car drives along the edge of a cliff but straight to the destination. This path takes less time but requires requires greater capabilities (both during and after) to avoid catastrophic outcomes. (Path 2): The car takes a longer path far away from the cliff, reducing risk. The dotted lines indicate different possible paths that could occur when executing a post-RLHF policy: red indicates the possibility that the agent may drive off the cliff if it fails to perfectly imitate Path 1; purple indicates a suboptimal policy starting from the final state of Path 1 that also drives off the cliff; green indicates a policy that is able to safely reach the goal. If the post-RLHF policy would induce the red or purple trajectories under a Path 1 preference, then it would be better for the labeler to have preferred Path 2, despite the longer travel time.
Refer to caption: extracted/6502829/figures/agentcapabilityfig.png

However, even the regret-based approach may not fully capture the factors that humans may consider when giving preferences. The regret model assumes that humans judge the “goodness” of an action as a function of the optimal advantage function $A^{*}$ that the agent should ideally be able to achieve under the assumed MDP, given enough data.
But what if humans do not assume optimality when judging action quality, but instead rely on their beliefs about the agent’s capabilities—in other words $A^{\pi_{\textrm{belief}}}$ for some imagined policy $\pi_{\textrm{belief}}$ rather than $A^{*}$?

For example, consider the scenario in Figure [1](https://arxiv.org/html/2506.01692v1#S1.F1), in which an agent must choose between a shorter cliff-side path to a goal or a longer, safer path far away from the cliff. Path 1 is preferable if the agent can reproduce the trajectory noiselessly and also behave optimally afterwards. However, if the agent is suboptimal, then it risks falling off the cliff when taking this path or afterwards. Conversely, Path 2 is better for a suboptimal agent, but overly conservative for an optimal agent. Suppose that a human labeler provides a preference for Path 1 based on the incorrect belief that the agent will behave optimally post-RLHF. If the agent is trained on these preferences and is suboptimal post-RLHF, this can lead to catastrophic outcomes.

This motivates the questions: Should a rational preference labeler consider agent capabilities? And what is a normative ideal for labeler beliefs about agent capabilities? There are a variety of factors that can affect an agent’s capabilities. Factors such as limited training data, incomplete or noisy data, or policy parameterization could limit the space of learnable policies. This limitation of learnable policies, by extension, limits agent performance. Furthermore, if the agent is a robot, physical limitations may also limit agent performance beyond what a human might assume.
Given a fixed set of trajectory pairs, we show that the normative ideal in regret-based RLHF is for labelers to provide preferences with respect to a quantity that is related to the best possible post-RLHF policy under any labeling—as opposed to the standard assumption of absolute optimality.

Our contributions are as follows. (1) We illustrate the importance of beliefs about agent capabilities by showing an example where poor agent-labeler agreement leads to suboptimal post-RLHF policies.
(2) We define a normative ideal of beliefs and theoretically investigate the impact of deviations from this ideal on the expected return of the post-RLHF policy. (3) We empirically examine the effect of the magnitude of deviation from the ideal belief by manipulating it directly in synthetic experiments, showing that the highest performance typically results from beliefs closest to our theoretical normative ideal.
(4) Finally, we confirm in a human study that human beliefs about the agent’s capabilities affect preferences. Specifically we show that it is possible to change human preferences in a statistically significant manner by changing their beliefs through priming. This study also points to potential paths forward for aligning labeler beliefs with agent capabilities.

## 2 Preliminaries

A Markov Decision Process (MDP) (Puterman, 2014; Sutton & Barto, 1998) is specified by a tuple $(\mathcal{S},\mathcal{A},P,\gamma,\mu,r)$. Here, $\mathcal{S}$ is the set of possible states, and $\mathcal{A}$ is the set of all possible actions. $P$ is the transition function $P:\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S})$, a probability distribution over the next state given the current state and action. $\gamma$ is the discount factor, $\mu$ is the start state distribution, and $r:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}$ is the scalar reward function. $r_{t}$ refers to the reward received at time $t$ for taking action $a_{t}$ at state $s_{t}$ and reaching $s_{t+1}$. Terminal states have a reward of 0, unless otherwise specified, upon reaching them and a reward of 0 forever after.
Throughout this paper, $r$ refers to the ground-truth reward function, from which preferences are generated.

A policy $\pi:\mathcal{S}\rightarrow\Delta(\mathcal{A})$, specifies the probability of taking an action in a given state. $V^{\pi}_{r}$ is the the state-value function for $\pi$ under the reward function $r$, defined as: $V^{\pi}_{r}(s)=E_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t},s_{t+1})|s_{
0}=s]$. $Q^{\pi}_{r}$ is the state-action value function: $Q^{\pi}_{r}(s,a)=E_{\pi}[r(s,a,s^{\prime})+\gamma V^{\pi}_{r}(s^{\prime})]$. The advantage function with respect to $\pi$ is defined as $A^{\pi}_{r}(s,a)=Q^{\pi}_{r}(s,a)-V^{\pi}_{r}$. $J^{\pi}_{r}=E_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t},s_{t+1})|s_{0}
\sim\mu]$ denotes the policy’s expected discounted return. We use $A^{\pi}$, $V^{\pi}$, $Q^{\pi}$, and $J^{\pi}$ to denote $A^{\pi}_{r}$, $V^{\pi}_{r}$, $Q^{\pi}_{r}$, and $J^{\pi}_{r}$ respectively.
An optimal policy $\pi^{*}$ is a policy for which $V^{\pi^{*}}_{r}(s)\geq V^{\pi}_{r}(s)$ for all states and all policies. We use $A^{*}$, $V^{*}$, and $Q^{*}$ to denote $A^{\pi^{*}}_{r}$, $V^{\pi^{*}}_{r}$, and $Q^{\pi^{*}}_{r}$ respectively.

Trajectories: $\sigma$ refers to a trajectory with starting state $s^{\sigma}_{0}$. The length of a trajectory $|\sigma|$ refers to the number of transitions in the trajectory. Each trajectory has $|\sigma+1|$ states and $|\sigma|$ actions. In the context of our setting, trajectories do not contain any reward information as rewards are unknown and are to be inferred from preferences.

Preference Dataset: We have a preference dataset $\mathcal{D}_{\text{pref}}=\{(\sigma^{+}_{i},\sigma^{-}_{i})\mid i=1,2,...,n$} where each pair of trajectories $(\sigma^{+}_{i},\sigma^{-}_{i})$ are of equal length and $\sigma^{+}_{i}\succ\sigma^{-}_{i}$ denotes that the first trajectory is strictly preferred over the second.

Preference Models: The pairwise preferences in $\mathcal{D}_{\text{pref}}$ are assumed to be sampled from an underlying distribution $P(\sigma^{+}\succ\sigma^{-})$ which is modeled using preference model (e.g. the partial-return model (Christiano et al., 2017) or the regret model (Knox et al., 2024)) obeying ground truth reward $r$. The partial-return preference model is given by:

$$ $\displaystyle P(\sigma^{+}\succ\sigma^{-})=\frac{\exp(\alpha\sum_{{}_{\sigma^{ +}}}\gamma^{t}r_{t})}{\exp(\alpha\sum_{{}_{\sigma^{+}}}\gamma^{t}r_{t})+\exp( \alpha\sum_{{}_{\sigma^{-}}}\gamma^{t}r_{t})}$ (1) $$

and the regret-based preference model is given by :

$$ $\displaystyle P^{*}(\sigma^{+}\succ\sigma^{-})$ $\displaystyle=\frac{\exp(\alpha\sum_{\sigma^{+}}\gamma^{t}A^{*}(s_{t},a_{t}))} {\exp(\alpha\sum_{\sigma^{+}}\gamma^{t}A^{*}(s_{t},a_{t}))+\exp(\alpha\sum_{ \sigma_{-}}\gamma^{t}A^{*}(s_{t},a_{t}))}$ (2) $$

where $\alpha$ is an inverse temperature coefficient.

## 3 Related Work

### 3.1 Reinforcement Learning from Human Feedback

RLHF
methods traditionally follow a two-step learning framework: (1) learning a reward function using preference data and (2) using RL to perform policy optimization  (Christiano et al., 2017; Ouyang et al., 2022; Brown et al., 2019). Direct methods remove the RL step from this traditional approach by directly learning policies from preferences but make the same preference modeling assumptions as standard return-based RLHF  (Munos et al., 2024; Rafailov et al., 2024; Hejna et al., 2024).

### 3.2 Modeling human beliefs in learning

Prior work has aimed to model human beliefs in learning settings beyond RLHF.
Reddy et al. state that the reason demonstrators deviate from near-optimal actions is because of a misunderstanding of the environment dynamics and propose an algorithm to estimate these misunderstandings (2018).
Gong & Zhang assume that humans do not maintain a correct belief about dynamics when giving preferences and propose a method to infer both the reward function and the human’s beliefs about dynamics (2020). Marklund & Roy aim to separate goals from potentially incorrect human beliefs about environmental transition dynamics (2024).
Chan et al. demonstrate that human irrationality, when correctly modeled, can be an asset to reward inference (2021).

### 3.3 Biases in human data collection

While RLHF assumes that humans offering preferences do so rationally and in alignment with their long-term objectives, biases in human survey data have been studied across various fields of research (Kaufmann et al., 2023; Kahneman, 2003; Köszegi & Rabin, 2008). It is widely recognized that human decisions can be swayed by biases due to irrationality, incomplete, or inaccurate information (Gilovich T, 2002). Köszegi & Rabin state that mistakes are systematic and can be revealed by behavior and show that previously seen small amounts of non representative data can bias preferences (2008).

Previous psychology-aware RLHF literature investigate the impact of biasing effects on preferences. Lichtenstein & Slovic discuss how preferences can be formed at the same time as the process of preference elicitation (2006). Personality, emotions, social connections, and decision biases (such as decoy effects, serial position effects, anchoring effects, framing effects, and group think) can also affect preferences  (Tran et al., 2021; Atas et al., 2021). Ziegler et al. discuss the importance of alignment between researcher goals and the labeler’s labels on open ended summarization and sentiment analysis tasks(2019).

## 4 Theory: Effects of agent-labeler disagreement

In this section, we aim to bound the impact of agent-labeler disagreement on expected returns of an RLHF-trained policy. We first formally define the notions of agent capability belief and agent-labeler disagreement. We then present an example MDP where agent-labeler disagreement leads to agents learning suboptimal policies. Finally, we provide a formal bound on regret of the post-RLHF policy loss as a function of agent-labeler disagreement.

Under the regret-based model of preferences, the labeler is assumed to provide preferences based on the advantage of actions under the optimal advantage function $A^{*}$.
However, a real-world agent’s
performance after RLHF will not typically be optimal and instead will be limited by the amount of preference data, training time, or the learning algorithm itself.
We will show that if humans do, in fact, generate preferences via the regret preference model, it can lead to catastrophically bad post-RLHF policies. This is due to the assumption of optimality following the trajectory segments for which preferences are given. For example, in Figure [1](https://arxiv.org/html/2506.01692v1#S1.F1), the actual advantage of choosing Path 1 depends on whether or not the car will behave optimally in future time steps during execution.
Thus, when providing preferences, we posit that the quality of an action ought to be related to the agent’s post-RLHF advantage function, rather than assuming optimality.

Formally, we denote the labeler’s estimate of the agent’s post-RLHF policy as $\pi_{\text{belief}}$.
$Q^{\pi_{\text{belief}}}$ and $A^{\pi_{\text{belief}}}$ denote the corresponding state-action value function and advantage function respectively. We propose a new preference model, similar to the regret-based preference model, but which aims to explicitly capture how humans might incorporate such beliefs. We call this new preference model the belief-based preference model:

$$ $\displaystyle P^{\text{belief}}(\sigma^{+}\succ\sigma^{-})$ (3) $\displaystyle=\leavevmode\resizebox{433.62pt}{}{$\frac{\exp(\alpha\sum_{\sigma ^{+}}\gamma^{t}A^{\pi_{\text{belief}}}(s_{t},a_{t}))}{\exp(\alpha\sum_{\sigma^ {+}}\gamma^{t}A^{\pi_{\text{belief}}}(s_{t},a_{t}))+\exp(\alpha\sum_{\sigma_{- }}\gamma^{t}A^{\pi_{\text{belief}}}(s_{t},a_{t}))}$}$ $$

where $\alpha$ is an inverse temperature coefficient. The remainder of the paper will examine the theoretical and empirical consequences of humans providing belief-based preferences under this model, as well as establish evidence via a human study that human preferences are, in fact, influenced by beliefs about agent capabilities. In order to do so, we must first formally define agent capabilities, as well as agent-labeler disagreement.

We assume that the human has a belief about the capabilities of the agent, which they use to generate preferences. We define this belief over capabilities in terms of a Q-function that represents the quality of an imagined agent’s policy:

###### Definition 4.1 .

Agent capability belief is defined as the state-action value function, $Q^{\pi_{\text{belief}}}$, used to generate preferences in the belief-based preference model proposed in equation [3](https://arxiv.org/html/2506.01692v1#S4.E3), via the associated $A^{\pi_{\text{belief}}}$.

In other words, preferences are generated based on the assumption that, after the completion of the preferred partial trajectory, the agent will follow $\pi_{\text{belief}}$, whose state-action value function is $Q^{\pi_{\text{belief}}}$.

One might then ask: what $\pi_{\text{belief}}$ might a human imagine when providing preferences? The standard regret-based model assumes that the human imagines optimal behavior, but our human study in Section [6](https://arxiv.org/html/2506.01692v1#S6) demonstrates that preferences are subject to priming effects, indicating that the human’s beliefs about agent capabilities may be more closely related to the perceived current policy of the agent, or a prior over similar agents formed from past experiences, or perhaps an estimated performance ceiling for such agents.

Thus, another key question is: What is the ideal $Q^{\pi_{\text{belief}}}$ for a human to hold? Assume that the human is asked to provide preferences over a fixed, finite set of pairs of partial trajectories. The normative ideal is for the human to provide preferences in a manner that results in the highest expected return of the post-RLHF policy. We denote any policy that achieves this maximum as $\pi_{\text{post}}^{*}$. Note that this policy is not necessarily optimal for the MDP; it is simply the best that can be achieved after an RLHF step under any assignment of preferences over the fixed set of trajectory pairs. Further, there exist a set of beliefs that lead to such maximizing policies when preferences are generated according to them. We denote these as $Q^{\pi_{\text{belief}}^{*}}$. This optimal belief may require oracle knowledge to generate (e.g. environmental dynamics or learning rule), but it is the normative ideal.

Finally, note that if an infinite number of (noiseless) preferences are to be given and any policy is representable by the agent, this implies that the labeler ought to have a belief of optimal behavior, normatively; in this case, $\pi_{\text{post}}^{*}$ is only suboptimal when a finite number of preferences are given. However, if preferences are noisy or the agent cannot represent an optimal policy (e.g. a weak function approximator), then this may not be true.

We now define agent-labeler disagreement, which captures the more realistic scenario of a human labeler providing preferences under some agent capability belief $Q^{\pi_{\text{belief}}}$ that is different from normative ideal, $Q^{\pi_{\text{belief}}^{*}}$.

###### Definition 4.2 .

We define the agent-labeler disagreement at a state-action pair as the magnitude of difference between an optimal belief $Q^{\pi_{\text{belief}}^{*}}$ and a human labeler’s belief $Q^{\pi_{\text{belief}}}$:

$$ $|Q^{\pi_{\text{belief}}^{*}}(s,a)-Q^{\pi_{\text{belief}}}(s,a)|.$ $$

When there exist multiple $Q^{\pi_{\text{belief}}^{*}}$, the minimum disagreement is used.

###### Definition 4.1 .

###### Definition 4.2 .

### 4.1 A case study

Figure: Figure 2: An MDP in which a safe and a risky action are available. If $A^{\pi_{\text{belief}}}(s_{0},a_{\text{risk}})>A^{\pi_{\text{belief}}}(s_{0},a _{\text{safe}})$ when $A^{\pi_{\text{belief}}^{*}}(s_{0},a_{\text{risk}})\leq A^{\pi_{\text{belief}}^ {*}}(s_{0},a_{\text{safe}})$, labelers will erroneously prefer the partial trajectory $(s_{0},a_{\text{risk}},s_{\text{risk}})$ over $(s_{0},a_{\text{safe}},s_{\text{safe}})$ under the belief-based preference model in Equation [3](https://arxiv.org/html/2506.01692v1#S4.E3).

Here we present an example to illustrate the effect of agent-labeler disagreement. Consider the simple MDP shown in Figure [2](https://arxiv.org/html/2506.01692v1#S4.F2), which is adapted from an example in Knox et al. (2024) to capture the risk of poor decision-making after an action.(^1^11 Note that decision-making itself may not appear to be a risk from the agent’s perspective, since the agent controls its own actions at subsequent states. However, even deterministically chosen actions can be risky from the labeler’s perspective because the labeler is only able to influence the agent at the current state.)
$s_{0}$ is the start state, and arrows show deterministic transitions towards the three terminal states on the far right. All rewards are 0 unless otherwise noted.

Suppose a labeler is asked to provide preferences between two partial trajectories: $(s_{0},a_{\text{risk}},s_{\text{risk}})$ and $(s_{0},a_{\text{safe}},s_{\text{safe}})$, and their beliefs are such that $A^{\pi_{\text{belief}}}(s_{0},a_{\text{risk}})>A^{\pi_{\text{belief}}}(s_{0},a
_{\text{safe}})$.
Then, under the belief-based preference model, they would tend to label $(s_{0},a_{\text{risk}},s_{\text{risk}})\succ(s_{0},a_{\text{safe}},s_{\text{
safe}})$.
If the agent-labeler disagreement is such that $A^{\pi_{\text{belief}}^{*}}(s_{0},a_{\text{risk}})\leq A^{\pi_{\text{belief}}^
{*}}(s_{0},a_{\text{safe}})$—which would occur if $\pi_{\text{belief}}^{*}(s_{\text{risk}},a_{\text{lose}})>0.5$—then obeying the preference above results in losing more often than winning during execution, resulting in a learned policy that has lower expected return than taking $a_{\text{safe}}$.

Conversely, suppose the labeler’s beliefs are such that $A^{\pi_{\text{belief}}}(s_{0},a_{\text{risk}})<A^{\pi_{\text{belief}}}(s_{0},a
_{\text{safe}})$. Then under the belief-based preference model, they would prefer $(s_{0},a_{\text{safe}},s_{\text{safe}})$ over $(s_{0},a_{\text{risk}},s_{\text{risk}})$. Obeying this preference results in a return of 0 instead of a higher return during execution if $A^{\pi_{\text{belief}}^{*}}(s_{0},a_{\text{risk}})>A^{\pi_{\text{belief}}^{*}}
(s_{0},a_{\text{safe}})$. Thus, we can see that agent-labeler disagreements can lead to suboptimal preferences and resulting policies.

### 4.2 Formal bounds on impact of agent-labeler disagreements

We now bound the impact on the expected return of the RLHF-trained policy under agent-labeler disagreements. For simplicity, we will first examine a setting where the labeler is asked to provide a preference between trajectories under an agent-labeler disagreement at a single state-action pair. We later discuss how this analysis extends to multiple agent-labeler disagreements.

###### Theorem 4.3 .

Consider an RLHF setup where a human labeler is provided with a finite number of pairs of single-transition trajectories to label. Let $Q^{\pi_{\text{belief}}^{*}}$ be an optimal agent capability belief (i.e. the normative ideal belief) for generating preferences using belief-based model (Eqn [3](https://arxiv.org/html/2506.01692v1#S4.E3)) over these pairs of trajectories, and let $\pi_{\text{post}}^{*}$ be the corresponding post-RLHF policy.
Let $Q^{\pi_{\text{belief}}}$ be the human labeler’s belief with an agent-labeler disagreement of $\delta$ at a state-action pair $(s^{\prime},a^{\prime})$ w.r.t. $Q^{\pi_{\text{belief}}^{*}}$. Let the post-RLHF policy trained with preferences given under $Q^{\pi_{\text{belief}}}$ be $\pi_{\text{post}}^{\delta}$.
Assume that preferences are noiseless under the belief-based model (i.e. $\alpha\rightarrow\infty$).
Further assume that the policy is tabular and RLHF produces a deterministic policy that respects all the preferences. We then have:

$$ $J_{\pi_{\text{post}}^{\delta}}\geq J_{\pi_{\text{post}}^{*}}-\frac{\delta}{1- \gamma},$ (4) $$

where $J_{\pi_{\text{post}}^{*}}$ and $J_{\pi_{\text{post}}^{\delta}}$ are the expected returns of $\pi_{\text{post}}^{*}$ and $\pi_{\text{post}}^{\delta}$ respectively.

###### Proof.

Consider preferences given over single-transition trajectories via the belief-based model (Eqn. [3](https://arxiv.org/html/2506.01692v1#S4.E3)).
When RLHF respects the given preferences, resulting in a deterministic post-RLHF policy $\pi_{\text{post}}$, we have $\pi_{\text{post}}(s)=\operatorname*{arg\,max}_{a}Q^{\pi_{\text{belief}}}(s,a)$. Under perfect agent-labeler agreement, we are given the post-RLHF policy as $\pi_{\text{post}}^{*}$.

Now, we examine the impact of agent-labeler disagreement at $(s^{\prime},a^{\prime})$ on the post-RLHF policy. We have disagreement of

$$ $\delta=\Big{|}Q^{\pi_{\text{belief}}^{*}}(s^{\prime},a^{\prime})-Q^{\pi_{\text {belief}}}(s^{\prime},a^{\prime})\Big{|}.$ (5) $$

When this disagreement grows such that $\delta>Q^{\pi_{\text{belief}}}(s^{\prime},\pi_{\text{post}}^{*}(s^{\prime}))-Q
^{\pi_{\text{belief}}}(s^{\prime},a^{\prime})$, action $a^{\prime}$ becomes the most preferred action. This results in a new post-RLHF policy $\pi_{\text{post}}^{\delta}$ as follows:

$$ $\displaystyle\pi_{\text{post}}^{\delta}(s)=\begin{cases}\pi_{\text{post}}^{*}( s)&\text{if }s\neq s^{\prime}\\ a^{\prime}&\text{if }s=s^{\prime}.\end{cases}$ (6) $$

Following Kakade & Langford (2002) and Schulman et. at. (2015), we calculate the difference between $J_{\pi_{\text{post}}^{\delta}}$ and $J_{\pi_{\text{post}}^{*}}$.

$$ $\displaystyle J_{\pi_{\text{post}}^{\delta}}-J_{\pi_{\text{post}}^{*}}=\frac{1 }{1-\gamma}\mathbb{E}_{s\sim d^{\pi_{\text{post}}^{\delta}}_{\mu}}\mathbb{E}_{ a\sim\pi_{\text{post}}^{\delta}}\left[A^{\pi_{\text{post}}^{*}}(s,a)\right],$ (7) $$

where $d^{\pi_{\text{post}}^{\delta}}_{\mu}$ is the discounted steady-state distribution of the MDP under policy $\pi_{\text{post}}^{\delta}$ and start state distribution $\mu$, $d^{\pi_{\text{post}}^{\delta}}_{\mu}(s)=(1-\gamma)\sum_{t=0}^{\infty}\gamma^{t
}\mathbb{P}(s_{t}=s|\pi_{\text{post}}^{\delta},\mu)$, and $A^{\pi_{\text{post}}^{*}}(s,a)$ is the advantage function of policy $\pi_{\text{post}}^{*}$, $A^{\pi_{\text{post}}^{*}}(s,a)=Q^{\pi_{\text{post}}^{*}}(s,a)-V^{\pi_{\text{
post}}^{*}}(s)$.

We can further simplify equation [7](https://arxiv.org/html/2506.01692v1#S4.E7) considering the deterministic nature of $\pi_{\text{post}}^{*}$ and $\pi_{\text{post}}^{\delta}$ as:

$$ $\displaystyle J_{\pi_{\text{post}}^{\delta}}-J_{\pi_{\text{post}}^{*}}$ (8) $\displaystyle=\frac{1}{1-\gamma}\mathbb{E}_{s\sim d^{\pi_{\text{post}}^{\delta }}_{\mu}}\mathbb{E}_{a\sim\pi_{\text{post}}^{\delta}}\left[Q^{\pi_{\text{post} }^{*}}(s,a)-Q^{\pi_{\text{post}}^{*}}(s,\pi_{\text{post}}^{*}(s))\right]$ $\displaystyle=\frac{1}{1-\gamma}\mathbb{E}_{s\sim d^{\pi_{\text{post}}^{\delta }}_{\mu}}\left[Q^{\pi_{\text{post}}^{*}}(s,\pi_{\text{post}}^{\delta}(s))-Q^{ \pi_{\text{post}}^{*}}(s,\pi_{\text{post}}^{*}(s))\right]$ (9) $$

Note that the two policies differ only at a state $s^{\prime}$. This allows us to further simplify equation [9](https://arxiv.org/html/2506.01692v1#S4.E9) as:

$$ $\displaystyle=\frac{d^{\pi_{\text{post}}^{\delta}}_{\mu}(s^{\prime})}{1-\gamma }\left[Q^{\pi_{\text{post}}^{*}}(s^{\prime},\pi_{\text{post}}^{\delta}(s^{ \prime}))-Q^{\pi_{\text{post}}^{*}}(s^{\prime},\pi_{\text{post}}^{*}(s^{\prime }))\right]$ (10) $\displaystyle\geq\frac{d^{\pi_{\text{post}}^{\delta}}_{\mu}(s^{\prime})}{1- \gamma}\left[Q^{\pi_{\text{post}}^{*}}(s^{\prime},\pi_{\text{post}}^{*}(s^{ \prime}))-\delta-Q^{\pi_{\text{post}}^{*}}(s^{\prime},\pi_{\text{post}}^{*}(s^ {\prime}))\right]$ (11) $\displaystyle\geq\frac{d^{\pi_{\text{post}}^{\delta}}_{\mu}(s^{\prime})}{1- \gamma}\left(-\delta\right)$ (12) $\displaystyle\geq-\frac{\delta}{1-\gamma}\implies\boxed{J_{\pi_{\text{post}}^{ \delta}}\geq J_{\pi_{\text{post}}^{*}}-\frac{\delta}{1-\gamma}}$ (13) $$

The inequality [11](https://arxiv.org/html/2506.01692v1#S4.E11) comes from definition of agent-labeler disagreement. Inequality [13](https://arxiv.org/html/2506.01692v1#S4.E13) follows from the fact that $0\leq d^{\pi_{\text{post}}^{\delta}}_{\mu}(s)\leq 1$. This concludes the proof.

∎

Next, we show how this proof extends to multiple agent-labeler disagreements.

###### Corollary 4.4 .

When there are multiple agent-labeler disagreements, $\{\delta_{1},\delta_{2},\cdots,\delta_{k}\}$, the Theorem [4.3](https://arxiv.org/html/2506.01692v1#S4.Thmtheorem3) can be extended as:

$$ $J_{\pi_{\text{post}}^{\delta}}\geq J_{\pi_{\text{post}}^{*}}-\frac{\max\{ \delta_{1},\delta_{2},\cdots,\delta_{k}\}}{1-\gamma}$ $$

###### Proof.

Refer to Appendix [A](https://arxiv.org/html/2506.01692v1#A1) for the proof.
∎

###### Theorem 4.3 .

###### Proof.

###### Corollary 4.4 .

###### Proof.

## 5 Empirical Study of Impact of Agent-Labeler Disagreements on Policy Learning

In the prior section, we bounded the effects of agent-labeler disagreement for simple tabular policies without function approximation or generalization. In this section, we aim to empirically study this phenomenon for a broader class of policy representations that can exhibit generalization, as well as limit the set of representable policies. As noted earlier, when infinite noiseless preferences are given, then the normative ideal is for the labeler to give preferences with respect to an optimal policy, as long as the agent can represent and execute an optimal policy. However, if maximum agent performance is limited in some manner—for example, due to a weak function approximator, regularized learning rule, or noisy actuators that don’t faithfully execute the commanded action—then the normative ideal is to give preferences with respect to the best achievable policy.

To study the impact of agent-labeler disagreement in a more realistic setting in which the agent is limited,
we design an experiment using synthetic data in a gridworld domain shown in Figure [3](https://arxiv.org/html/2506.01692v1#S5.F3).
The agent can only represent epsilon-greedy policies for a particular value of epsilon. This policy noise is intended to serve as a stand-in for the various types of performance-capping limitations discussed above, and has the advantage of being easy to systematically manipulate. We model the labeler’s beliefs very simply as a belief over epsilon; they assume an optimal policy within the representable epsilon-greedy class. Thus, agent-labeler disagreement arises due to incorrect beliefs about the value of epsilon. We hypothesize that performance of the post-RLHF policy will degrade as the labeler’s belief over epsilon diverges from the true epsilon.

Figure: Figure 3: 7x7 GridWorld with start state (in yellow), two terminal cliff states (in red) and one terminal goal state (in green). In each cell, we mark the reward incurred for reaching the state.

We first collect 100 random trajectories (enough to guarantee a nearly optimal-within-class post-RLHF policy) of the agent traversing the gridworld environment that terminate either in the goal state or in the cliff state. We denote $\epsilon$ as the $\epsilon$-greedy noise impacting the post-RLHF policy of the agent and $\epsilon^{\prime}$ as the $\epsilon$-greedy noise assumed by the labeler and determining the labeler’s preferences. The values of $\epsilon$ and $\epsilon^{\prime}$ may be different.
We denote the labeler’s belief about agent’s capability $Q^{\pi_{\text{belief}}^{\epsilon^{\prime}}}(s,a)$. We denote the post-RLHF agent capability as $Q^{\pi_{\text{post}}^{\epsilon}}(s,a)$. We vary the values of $\epsilon$ and $\epsilon^{\prime}$ to show the impact of agent-labeler disagreements on the post-RLHF policy’s expected return.

We use $Q^{\pi_{\text{belief}}^{\epsilon^{\prime}}}(s,a)$ to generate synthetic preference data using the using the belief-based preference model shown in Equation [3](https://arxiv.org/html/2506.01692v1#S4.E3). We use these preferences to train the agent using using Contrastive Preference Learning (CPL) (Hejna et al., 2024), a scalable algorithm for regret-based RLHF. See Appendix [C.2](https://arxiv.org/html/2506.01692v1#A3.SS2) for the hyperparameters used in this experiment.

Table: Table 1: The effect of mismatch in belief in the agent capabilities (rows) and the agent’s actual capability (columns). Highest returns generally correspond to the most aligned beliefs, shown along the diagonal. The error intervals show the 95% confidence interval.

Table [1](https://arxiv.org/html/2506.01692v1#S5.T1) shows the average returns of the post-RLHF policies. The highest return generally comes when the $\epsilon^{\prime}$-greedy noise assumed by labeler is closest to the $\epsilon$-greedy noise impacting the agent’s policy. We see that for lower values of epsilon ($\epsilon=0\;\&\;0.1$), preferences given under the optimal advantage function ($\epsilon^{\prime}=0$) provide the highest returns. However, this strategy becomes a hindrance for effective RLHF as the action noise increases, as seen when $\epsilon=0.3\;\&\;0.5$. This is due to the fact that the post-RLHF policy learns to walk near the dangerous cliffs, often falling off, since the labeler assumed too low of a noise level. Walking near the cliffs is only optimal when the agent has a very low epsilon value.
These results serve as a proof of concept for how agent-labeler disagreements can arise in practice and how they can be exacerbated by agent limitations that may be unknown to the labeler.

## 6 Agent-Labeler Disagreements in Human Preference Collection

Now that we have established that agent-labeler disagreements pose a problem in RLHF, in this section, we demonstrate that human beliefs about agent capabilities do, in fact, significantly affect the preferences given by human labelers. This is done via a human study in which priming effects are used change participants’ beliefs about the agent’s capabilities. Additionally, this result indicates that it is possible in theory to change human preferences to be better aligned with agent capabilities.

Domain: We choose a self-driving car domain, simulated using the CARLA driving simulator (Dosovitskiy et al., 2017) to collect trajectories to use for human preference collection. See Appendix [B.1](https://arxiv.org/html/2506.01692v1#A2.SS1) for additional details. Since the rules and trade-offs involved in driving are something that many people already have a strong intuition for, the choice of self-driving car setup allows us to keep the filtering of the participants minimal. We use Prolific(^2^22[https://www.prolific.com/](https://www.prolific.com/)) for preference collection. Additional details on the demographics of the participants are provided in Appendix [B.4](https://arxiv.org/html/2506.01692v1#A2.SS4).

Priming: Before beginning data collection, we randomly assign participants to one of three conditions: (1) no priming (control condition), (2) safe priming (confidence increasing condition), and (3) unsafe priming (confidence decreasing condition).

Both safe and unsafe priming videos start from the same starting state to avoid unintended biasing effects.
Participants in the safe priming group are shown a video where the car obeys all traffic laws, avoids obstacles, and successfully overtakes a slower car, whereas the participants in the unsafe priming group are shown a video where the car drifts across lanes and crashes into obstacles.

Preference elicitation: Participants are asked to provide preferences over 15 randomly chosen pairs of video data. All data pairs on which preferences are collected show legal driving. Within each pair, the car is shown to be driving on the same path for the same amount of time. In each pair, one video shows faster and more time saving behavior, which if executed imperfectly (during or after the trajectory) could lead to failure states, such as hitting a pedestrian, another car, or breaking traffic laws; the other shows more safety-conscious behavior that requires more time to execute, but for which it is easier to avoid failure states. See Appendix [B.3](https://arxiv.org/html/2506.01692v1#A2.SS3) for an example trajectory. Each pair of trajectories is designed to make the participants consider both: (a) how a learning agent might behave in the same states shown in the trajectories and (b) what the agent might do after the end of each trajectory.

All data pairs are randomly shuffled, and the order in which each pair of data is shown is also randomly shuffled to avoid ordering bias during preference collection. In each question, both videos in each pair are presented next to each other. Figure [5](https://arxiv.org/html/2506.01692v1#A2.F5) in Appendix [B.2](https://arxiv.org/html/2506.01692v1#A2.SS2) shows an example questionnaire shown to participants for a single pair of trajectories.

Instructions for the participants:
Participants in the control group are given a standard preference collection instruction explaining that the objective of the survey is to improve a self-driving car using their preferences over behavioral data. We chose to frame this language for this group to indicate that their preferences were not meant to be a rating of the trajectories themselves, but rather a learning signal for improving a self driving car’s policy. This was designed to lead them to reflect less on the trajectories themselves and more on their beliefs about how a learning agent might behave in similar or future states. The participants in the safe and unsafe priming groups are given a similar instruction with an addition of a video showcasing the car’s driving behaviors.
To encourage the participants to think deeply about car’s capabilities, they are then asked about which skills they think would be useful for the car to learn to become a more effective driver.

Post-processing of collected data: To maintain a high standard while finalizing the data, we construct a participant filtering step that ensures that participants are paying attention when providing preferences. First, we show a pair of videos: one in which the car drives normally and one in which the car crashes into a guardrail while making a turn. Additionally, we show a pair of videos in which one car drives normally and the other runs a red light. We remove all preference data from participants who do not strongly prefer the attention check videos that show good driving, while also reporting that they were extremely confident.

In total, we collected data from 259 participants. The data filtering and quality checks allowed 146 (46 safe priming, 50 unsafe priming, 50 no priming) participants’ data to be used for analysis. We further balanced the data in each group to avoid unintended impacts on the statistical analysis by randomly removing 4 entries from both the unsafe and no priming groups. The final dataset contains 46 participants in each of the three groups.

To the best of our knowledge, there is not a standard nonparametric statistical test that can test for statistical significance when there are: (1) more than two groups; (2) for which the data is Likert; (3) and for which each participant gives repeated measures data. Instead, we used the average of each participant’s responses, rather than individual question responses, to create independent data that is amenable to standard statistical tests, as further described in the next subsection. To reduce noise due to participant confusion, uncertainty, or misunderstandings, we used only those responses where participants reported extremely high confidence in their answers.

Qualitative results:
We asked participants to provide text feedback on the car’s performance immediately after priming in order to check their beliefs after priming. We used GPT-4 (OpenAI et al., 2024) to conduct a sentiment analysis for each response and provide a 1-10 star rating. We prompted GPT-4 to take into account safety and other necessary skills in the rating. We use the following prompt to get the ratings for the priming texts:

Analyze the following feedback about a car’s driving skills and rate its safety from 1 to 10 stars. Are there any skills this car need to learn to be a more effective driver? Only consider the presence of any safety concerns and suggestions for improvement with small consideration for factors unrelated to safety: Feedback: (PARTICIPANT’S FEEDBACK) Provide a structured analysis with reasons for the assigned score.

Figure [4](https://arxiv.org/html/2506.01692v1#S6.F4) shows that immediately after priming, participants primed with the unsafe priming had lower sentiment scores regarding the car’s capabilities than the participants in the safe priming group, suggesting that priming had an impact on their beliefs.

Figure: Figure 4: Sentiment analysis on participants’ written responses about the car’s capabilities after priming, evaluated on a 1–10 scale by GPT-4. Histogram bars for both groups are plotted from 0, not stacked.
Refer to caption: extracted/6502829/figures/gpt4sentiment.png

Quantitative results:
Next, we analyzed the Likert data associated with the preferences of each condition group.
While some research suggests that ANOVA may be suitable for Likert data, it is still a controversial test to use for this purpose and has been shown to lose power when used for non-metric data (Eiselen & van Huyssteen, 2023). We used the non-parametric Kruskal-Wallis test for statistical analysis due to the ordinal nature of the data (Kruskal & Wallis, 1952). The Kruskal-Wallis test resulted in a p-value of 0.033, indicating a statistically significant difference between at least two priming groups at the 5% level. In order to detect which of the two priming groups differ, we used the Dunn’s test with Bonferroni correction (Dunn, 1961). Dunn’s test is a non-parametric pairwise comparison test that controls for the Type 1 error that can occur when multiple comparisons are made simultaneously. Dunn’s test offers multiple corrections to control for Type 1 error, of which the Bonferroni correction is the most conservative. The results of the Dunn-Bonferroni test are shown in Table [2](https://arxiv.org/html/2506.01692v1#S6.T2) and show statistically significant ($p=0.015$) differences between safe and unsafe priming groups at the 5% level. Cliff’s delta (d = 0.31) between unsafe vs safe priming shows that participants primed with unsafe priming were influenced toward more safe responses at a higher rate than participants primed with safe priming with a small effect size (Cliff, 1993). Participants in the no priming group did not exhibit statistically significant differences in their responses compared to either the unsafe or safe priming groups, as shown in Table [2](https://arxiv.org/html/2506.01692v1#S6.T2).

Table: Table 2: P-values for Dunn-Bonferroni for responses given with extremely strong confidence. P-values below 0.05 indicate statistical significance at the 5% level.

These results suggest that human preferences are influenced by their beliefs about agent capabilities. To mitigate this bias, practitioners should actively attempt to strengthen agent-labeler agreement during data collection.

### 6.1 Recommendations For Researchers

Our results show that having good agent-labeler agreement is important for effective policy learning in regret-based RLHF. While it is difficult to reason about an agent’s post-training performance prior to training, our results suggest potential best practices for practitioners:

Inform labelers directly about known limitations: Practitioners may choose to inform labelers about agent limitations prior to preference collections. For example, this may include known data or training limitations, non-obvious restrictions on agent capabilities (e.g. an unusually limited turning radius on a vehicle), or assessments of projected agent strengths and weaknesses. However, the success of such an approach depends heavily on the ability of labelers to make sense of such information, which may require varying levels of technical expertise.

Online preference collection with intermittent priming:
Practitioners may choose to use online preference collection interleaved with RLHF training on the collected preferences. Here, labelers’ beliefs about the agent’s capability may be continuous updated and aligned via priming using data from the agent’s most current policy. This approach may also demonstrate to labelers how their preferences influence policy learning.

## 7 Conclusion

Current methods of preference collection do not account for participants’ beliefs about the capabilities of the agents learning from their preferences. We show that the preferences collected under agent-labeler disagreements can lead to suboptimal policies both theoretically and empirically and make preliminary suggestions on how to minimize such disagreements. Future work may include developing improved priming strategies to ensure human beliefs are as accurate as possible, as well as algorithmic advances that mitigate the impact of incorrect beliefs on RLHF.

## References

- Amodei et al. (2016)
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D.
Concrete problems in ai safety, 2016.
URL [https://arxiv.org/abs/1606.06565](https://arxiv.org/abs/1606.06565).
- Atas et al. (2021)
Atas, M., Felfernig, A., Polat-Erdeniz, S., Popescu, A., Tran, T. N. T., and Uta, M.
Towards psychology-aware preference construction in recommender systems: Overview and research issues.
*J. Intell. Inf. Syst.*, 57(3):467–489, December 2021.
ISSN 0925-9902.
doi: 10.1007/s10844-021-00674-5.
URL [https://doi.org/10.1007/s10844-021-00674-5](https://doi.org/10.1007/s10844-021-00674-5).
- Brown et al. (2019)
Brown, D. S., Goo, W., Nagarajan, P., and Niekum, S.
Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations.
In *International Conference on Machine Learning*, 2019.
URL [https://api.semanticscholar.org/CorpusID:119111734](https://api.semanticscholar.org/CorpusID:119111734).
- Chan et al. (2021)
Chan, L., Critch, A., and Dragan, A.
Human irrationality: both bad and good for reward inference, 2021.
URL [https://arxiv.org/abs/2111.06956](https://arxiv.org/abs/2111.06956).
- Christiano et al. (2017)
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D.
Deep reinforcement learning from human preferences.
*Advances in neural information processing systems*, 30, 2017.
- Cliff (1993)
Cliff, N.
Dominance statistics: Ordinal analyses to answer ordinal questions.
*Psychological Bulletin*, 114:494–509, 11 1993.
doi: 10.1037/0033-2909.114.3.494.
- Dosovitskiy et al. (2017)
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., and Koltun, V.
Carla: An open urban driving simulator.
In *Conference on robot learning*, pp.  1–16. PMLR, 2017.
- Dunn (1961)
Dunn, O. J.
Multiple comparisons among means.
*Journal of the American statistical association*, 56(293):52–64, 1961.
- Eiselen & van Huyssteen (2023)
Eiselen, R. and van Huyssteen, G. B.
A comparison of statistical tests for likert-type data: The case of swearwords.
*Journal of Open Humanities Data*, Oct 2023.
doi: 10.5334/johd.132.
- Gilovich T (2002)
Gilovich T, Griffin D, K. D.
*Heuristics and Biases: The Psychology of Intuitive Judgment*.
Cambridge University Press, 2002.
- Gong & Zhang (2020)
Gong, Z. and Zhang, Y.
What is it you really want of me? generalized reward learning with biased beliefs about domain dynamics.
*Proceedings of the AAAI Conference on Artificial Intelligence*, 34:2485–2492, 04 2020.
doi: 10.1609/aaai.v34i03.5630.
- Harris et al. (2020)
Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M. H., Brett, M., Haldane, A., del Río, J. F., Wiebe, M., Peterson, P., Gérard-Marchant, P., Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H., Gohlke, C., and Oliphant, T. E.
Array programming with NumPy.
*Nature*, 585(7825):357–362, September 2020.
doi: 10.1038/s41586-020-2649-2.
URL [https://doi.org/10.1038/s41586-020-2649-2](https://doi.org/10.1038/s41586-020-2649-2).
- Hejna et al. (2024)
Hejna, J., Rafailov, R., Sikchi, H., Finn, C., Niekum, S., Knox, W. B., and Sadigh, D.
Contrastive preference learning: Learning from human feedback without reinforcement learning.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=iX1RjVQODj](https://openreview.net/forum?id=iX1RjVQODj).
- Ji et al. (2023)
Ji, J., Qiu, T., Chen, B., Zhang, B., Lou, H., Wang, K., Duan, Y., He, Z., Zhou, J., Zhang, Z., et al.
Ai alignment: A comprehensive survey.
*arXiv preprint arXiv:2310.19852*, 2023.
- Kahneman (2003)
Kahneman, D.
Maps of bounded rationality: Psychology for behavioral economics.
*The American Economic Review*, 93(5):1449–1475, 2003.
ISSN 00028282.
URL [http://www.jstor.org/stable/3132137](http://www.jstor.org/stable/3132137).
- Kakade & Langford (2002)
Kakade, S. M. and Langford, J.
Approximately optimal approximate reinforcement learning.
In *International Conference on Machine Learning*, 2002.
URL [https://api.semanticscholar.org/CorpusID:31442909](https://api.semanticscholar.org/CorpusID:31442909).
- Kaufmann et al. (2023)
Kaufmann, T., Weng, P., Bengs, V., and Hüllermeier, E.
A Survey of Reinforcement Learning from Human Feedback, 2023.
- Knox et al. (2024)
Knox, W. B., Hatgis-Kessell, S., Booth, S., Niekum, S., Stone, P., and Allievi, A. G.
Models of human preference for learning reward functions.
*Transactions on Machine Learning Research*, 2024.
ISSN 2835-8856.
URL [https://openreview.net/forum?id=hpKJkVoThY](https://openreview.net/forum?id=hpKJkVoThY).
- Kruskal & Wallis (1952)
Kruskal, W. and Wallis, W.
Use of ranks in one-criterion variance analysis.
*Journal of the American Statistical Association*, pp.  583–621, 1952.
- Köszegi & Rabin (2008)
Köszegi, B. and Rabin, M.
*Revealed Mistakes and Revealed Preferences*, pp.  193–209.
Oxford Academic, 04 2008.
ISBN 9780195328318.
doi: 10.1093/acprof:oso/9780195328318.003.0008.
- Lichtenstein & Slovic (2006)
Lichtenstein, S. and Slovic, P.
*The Construction of Preference*.
Cambridge University Press, 2006.
- Marklund & Roy (2024)
Marklund, H. and Roy, B. V.
Choice between partial trajectories: Disentangling goals from beliefs, 2024.
URL [https://arxiv.org/abs/2410.22690](https://arxiv.org/abs/2410.22690).
- Munos et al. (2024)
Munos, R., Valko, M., Calandriello, D., Gheshlaghi Azar, M., Rowland, M., Guo, Z. D., Tang, Y., Geist, M., Mesnard, T., Fiegel, C., Michi, A., Selvi, M., Girgin, S., Momchev, N., Bachem, O., Mankowitz, D. J., Precup, D., and Piot, B.
Nash learning from human feedback.
In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), *Proceedings of the 41st International Conference on Machine Learning*, volume 235 of *Proceedings of Machine Learning Research*, pp.  36743–36768. PMLR, 21–27 Jul 2024.
URL [https://proceedings.mlr.press/v235/munos24a.html](https://proceedings.mlr.press/v235/munos24a.html).
- OpenAI et al. (2024)
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapiro, G., Berner, C., Bogdonoff, L., Boiko, O., Boyd, M., Brakman, A.-L., Brockman, G., Brooks, T., Brundage, M., Button, K., Cai, T., Campbell, R., Cann, A., Carey, B., Carlson, C., Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B., Cho, C., Chu, C., Chung, H. W., Cummings, D., Currier, J., Dai, Y., Decareaux, C., Degry, T., Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffet, A., Eleti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S. P., Forte, J., Fulford, I., Gao, L., Georges, E., Gibson, C., Goel, V., Gogineni, T., Goh, G., Gontijo-Lopes, R., Gordon, J., Grafstein, M., Gray, S., Greene, R., Gross, J., Gu, S. S., Guo, Y., Hallacy,
C., Han, J., Harris, J., He, Y., Heaton, M., Heidecke, J., Hesse, C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu, K., Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Jonn, B., Jun, H., Kaftan, T., Łukasz Kaiser, Kamali, A., Kanitscheider, I., Keskar, N. S., Khan, T., Kilpatrick, L., Kim, J. W., Kim, C., Kim, Y., Kirchner, J. H., Kiros, J., Knight, M., Kokotajlo, D., Łukasz Kondraciuk, Kondrich, A., Konstantinidis, A., Kosic, K., Krueger, G., Kuo, V., Lampe, M., Lan, I., Lee, T., Leike, J., Leung, J., Levy, D., Li, C. M., Lim, R., Lin, M., Lin, S., Litwin, M., Lopez, T., Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning, S., Markov, T., Markovski, Y., Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S. M., McLeavey, C., McMillan, P., McNeil, J., Medina, D., Mehta, A., Menick, J., Metz, L., Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, E., Mossing, D., Mu, T., Murati, M., Murk, O., Mély, D., Nair, A., Nakano, R.,
Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang, L., O’Keefe, C., Pachocki, J., Paino, A., Palermo, J., Pantuliano, A., Parascandolo, G., Parish, J., Parparita, E., Passos, A., Pavlov, M., Peng, A., Perelman, A., de Avila Belbute Peres, F., Petrov, M., de Oliveira Pinto, H. P., Michael, Pokorny, Pokrass, M., Pong, V. H., Powell, T., Power, A., Power, B., Proehl, E., Puri, R., Radford, A., Rae, J., Ramesh, A., Raymond, C., Real, F., Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H., Schnurr, D., Schulman, J., Selsam, D., Sheppard, K., Sherbakov, T., Shieh, J., Shoker, S., Shyam, P., Sidor, S., Sigler, E., Simens, M., Sitkin, J., Slama, K., Sohl, I., Sokolowsky, B., Song, Y., Staudacher, N., Such, F. P., Summers, N., Sutskever, I., Tang, J., Tezak, N., Thompson, M. B., Tillet, P., Tootoonchian, A., Tseng, E., Tuggle, P., Turley, N., Tworek, J., Uribe, J. F. C., Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C., Wang,
J. J., Wang, A., Wang, B., Ward, J., Wei, J., Weinmann, C., Welihinda, A., Welinder, P., Weng, J., Weng, L., Wiethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L., Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S., Yu, K., Yuan, Q., Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T., Zhuang, J., Zhuk, W., and Zoph, B.
Gpt-4 technical report, 2024.
URL [https://arxiv.org/abs/2303.08774](https://arxiv.org/abs/2303.08774).
- Ouyang et al. (2022)
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R.
Training language models to follow instructions with human feedback.
In *Proceedings of the 36th International Conference on Neural Information Processing Systems*, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc.
ISBN 9781713871088.
- Puterman (2014)
Puterman, M. L.
*Markov decision processes: discrete stochastic dynamic programming*.
John Wiley & Sons, 2014.
- Python Software Foundation (2023)
Python Software Foundation.
Python: Version 3.11, 2023.
URL [https://www.python.org/](https://www.python.org/).
- Rafailov et al. (2024)
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C.
Direct preference optimization: your language model is secretly a reward model.
In *Proceedings of the 37th International Conference on Neural Information Processing Systems*, NIPS ’23, Red Hook, NY, USA, 2024. Curran Associates Inc.
- Reddy et al. (2018)
Reddy, S., Dragan, A., and Levine, S.
Where do you think you're going?: Inferring beliefs about dynamics from behavior.
In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), *Advances in Neural Information Processing Systems*, volume 31. Curran Associates, Inc., 2018.
URL [https://proceedings.neurips.cc/paper_files/paper/2018/file/6f2268bd1d3d3ebaabb04d6b5d099425-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2018/file/6f2268bd1d3d3ebaabb04d6b5d099425-Paper.pdf).
- Schulman et al. (2015)
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P.
Trust region policy optimization.
In *International conference on machine learning*, pp.  1889–1897. PMLR, 2015.
- Sutton & Barto (1998)
Sutton, R. S. and Barto, A. G.
*Reinforcement Learning: An Introduction*.
MIT Press, Cambridge, MA, 1998.
- Tran et al. (2021)
Tran, T. N. T., Felfernig, A., and Tintarev, N.
Humanized recommender systems: State-of-the-art and research issues.
*ACM Transactions on Interactive Intelligent Systems*, 11:1–41, 07 2021.
doi: 10.1145/3446906.
- Virtanen et al. (2020)
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S. J., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., Carey, C. J., Polat, İ., Feng, Y., Moore, E. W., VanderPlas, J., Laxalde, D., Perktold, J., Cimrman, R., Henriksen, I., Quintero, E. A., Harris, C. R., Archibald, A. M., Ribeiro, A. H., Pedregosa, F., van Mulbregt, P., and SciPy 1.0 Contributors.
SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.
*Nature Methods*, 17:261–272, 2020.
doi: 10.1038/s41592-019-0686-2.
- Ziegler et al. (2019)
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G.
Fine-tuning language models from human preferences.
*ArXiv*, abs/1909.08593, 2019.
URL [https://api.semanticscholar.org/CorpusID:202660943](https://api.semanticscholar.org/CorpusID:202660943).

## Appendix A Proofs

Proof of the corollary [4.4](https://arxiv.org/html/2506.01692v1#S4.Thmtheorem4):

###### Proof.

Theorem [4.3](https://arxiv.org/html/2506.01692v1#S4.Thmtheorem3) states that the error bound on the expected return of a post-RLHF policy learned with preferences given under a single agent-labeler agreement at state $(s^{\prime},a^{\prime})$ of magnitude $\delta$ is:

$$ $J_{\pi_{\text{post}}^{\delta}}\geq J_{\pi_{\text{post}}^{*}}-\frac{\delta}{1- \gamma}.$ $$

This result can be interpreted as, in the worst case, the agent takes a sub-optimal action of magnitude $\delta$ at state $s^{\prime}$ from the start of the episode until the end of horizon.
Consider the case of multiple disagreements where we have state-action pair $(s^{\prime\prime},a^{\prime\prime})$ where we have disagreement of magnitude $\max\{\delta_{1},\delta_{2},...,\delta_{k}\}$ . In the worst case, the agent will be stuck at state $(s^{\prime\prime},a^{\prime\prime})$ from the beginning of the episode until of the horizon. This results in following bound on expected returns:

$$ $J_{\pi_{\text{post}}^{\delta}}\geq J_{\pi_{\text{post}}^{*}}-\frac{\max\{ \delta_{1},\delta_{2},...,\delta_{k}\}}{1-\gamma}.$ $$

∎

###### Proof.

## Appendix B Additional Details About Human Study

In this section, we provide more details about our human study and the demographics of participants.

### B.1 Design Decisions

We chose self-driving cars as the domain on which to collect our preference dataset because the rules and trade-offs of driving are something people understand without high level knowledge about any language or formal higher education and allowed for minimal filtering of participants. We conducted these preference collection experiments with prior IRB approval.

### B.2 Data Collection UI

Figure [5](https://arxiv.org/html/2506.01692v1#A2.F5) shows the format of the survey that participants saw when giving their responses.

Figure: Figure 5: The format of questions asked to participants in the survey. In order to avoid order biasing, we avoided terms such as “first/second”, “left/right”, or arrows and opted to use labels to indicate which side they prefer more. Subjects are also asked to report confidence in their response and a reasoning for their response. Subjects are required to respond to all three questions for each preference pair before they are allowed to move to the next pair in order to avoid confusion.
Refer to caption: extracted/6502829/figures/regretui2.png

### B.3 Example of a Pair of Trajectories Shown to Human Labelers

All trajectories shown to participants show legal driving. Each pair of trajectories show the car driving along the same route for the same amount of time so participants can easily distinguish the tradeoff between safety and time savings. For example, one pair of scenarios shows a legal overtake of a slow car in a single lane road, as shown in Figure [6](https://arxiv.org/html/2506.01692v1#A2.F6). While it is legal and safe to perform the overtake, doing so requires trust in the car’s abilities to successfully return back to its lane without any accidental swerving or any other mishaps. The pair shows one trajectory where the car does not attempt this overtake and instead patiently stays behind the slow car, as shown in Figure [6(a)](https://arxiv.org/html/2506.01692v1#A2.F6.sf1); the other shows the car overtaking successfully and getting further along the road in the same amount of time, as shown in Figure [6(b)](https://arxiv.org/html/2506.01692v1#A2.F6.sf2). This allows room for the subject to interpret the quality of the car’s driving for the overtake and be subjective in their belief in the car’s abilities.

Figure: (a) An example of a trajectory showing safety-conscious driving behavior. Here, the car chooses to stay behind the slower driver and finishes the trajectory near the start of the bridge.
Refer to caption: extracted/6502829/figures/carlasafetyvideo.png

### B.4 Demographics

Figure [7(a)](https://arxiv.org/html/2506.01692v1#A2.F7.sf1) shows the distribution of driver experience among the participants of our survey and Figure [7(b)](https://arxiv.org/html/2506.01692v1#A2.F7.sf2) shows participant’s willingness to ride in a self driving car. While we did not have specific filters for years of driving experience, most of the participants report a significant number of years of driving experience, meaning that they are well versed in the domain. A large portion of participants from a three priming groups reported that they would be willing to ride in a self driving car. However, out of all participants who reported that they would be unwilling to ride in a self driving car, the largest group of participants was the unsafe priming group.

Figure: (a) Participant’s driving years of experience.
Refer to caption: extracted/6502829/figures/experience.png

## Appendix C GridWorld Experimental Details

In this section, we provide additional details about the GridWorld domain and the hyperparameters used in the learning algorithm. For this analysis, we used the following python libraries: numpy (Harris et al., 2020), math and random from the standard python library (Python Software Foundation, 2023), and scipy(Virtanen et al., 2020).

### C.1 Gridworld Domain

The gridworld domain consists of a 7x7 grid of cells. We number the coordinates from 0 to 6. The agent always starts in the cell numbered (6,6). From there, the agent may move in any of the four cardinal directions. Any movement by the agent that results in the agent moving outside the boundaries of the environment results in the agent staying in its original state with an additional reward of -1. The states (0, 6), (3, 6), and (4, 6) are terminal states and result in ac reward of +200, -200, and -200 respectively. The episode terminates when the agent either reaches one of the terminal states or when the number of states that the agent has visited reaches 1,000.

### C.2 Hyperparameters

Here, we will discuss the hyperparameters used in the gridworld experiment for the Contrastive Preference learning algorithm:

- 1.
Discount Factor: 0.7
- 2.
Regularization ($\gamma$): 0.01
- 3.
Learning Rate: 0.5
- 4.
Temperature parameter ($\alpha$): 10
- 5.
Number of random seeds: 20
- 6.
Epochs: 20

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses above.
soundness: 4
strengths: - The paper introduces  a belief based model where the preferences over trajectories is proportional to the differences between advantages of the trajectories under the belief policy (while prior works assume an optimal policy). The anecdotal example is very intuitive and clearly presents the problem.
- The authors present a theoretical lower bound on the policy performance as the annotators deviate from the optimal beliefs. This misalignment leads to sub-optimal performance as shown in the grid based experiments.
- The problem of  annotator beliefs affecting preference labels is verified via a large human study, where the different groups are put under different beliefs about the driving agents capabilities and then are asked to rate behavior, clearly showing that if annotators have belief that the agent is risky, they prefer the conservative and safe behavior.
summary: The paper proposes a problem : human preferences are guided by beliefs over agent capabilities rather than only by the actions and observed outcomes. First, the authors verify this claim showing that if the annotators make incorrect assumptions about the policies, it leads to sub-optimal performance post RLF. The paper theoretically and experimental models the ideal belief to achieve maximum performance and then shows the impact of deviation from this ideal belief  on the expected return of the policy. Finally, a human study is conducted to provide stronger evidence for the claim that agent capability influences preference label;. Overall, the paper is very interesting to read, is well written and I strongly recommend accepting it.
weaknesses: - The paper lacks experiments at scale w.r.t to policy learning to demonstrate the effectiveness of the modelling agent belief capabilities. For eg. in the CARLA experiments, it would be interesting to show the return and qualitative performance of the agent when trained using preferences from different priming groups. This would show that the same policy under different beliefs converges to different performances (but a scalable experiment for the same).
- It would be interesting to hear the authors views on how this could also be incorporated into population based annotations and how the beliefs across a population affect each other? This seems to be a stronger source of misalignment than function approximation and limited data as overparameterized neural networks and LLM alignment being a bandit problem don't suffer from it as much.
- While the paper focuses on introducing the problem clearly, I request the authors to also include some insight towards how to build potential solutions towards handling the belief gap. This would make the paper even stronger and influential towards a wider audience.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Theorem 4.3 assumes noiseless preferences, tabular deterministic policies, and exact policy consistency with all preferences. How robust is the bound to realistic noise and approximation error in RLHF (e.g., function approximators, stochastic policies, non-deterministic preference datasets)? Could you provide an intuition or empirical simulation showing whether the derived bound approximately holds when these assumptions are relaxed?
2. How should we interpret $\pi_{belief}$ in practice? Is it meant to represent the human’s subjective prior over agent behavior or their estimate of the post-RLHF policy? Please clarify how you envision labelers forming this belief during annotation and how it might be approximated or measured in real settings.
3. In the human study, you infer that priming alters participants’ beliefs about agent capabilities, but you do not measure those beliefs directly. Have you considered collecting explicit quantitative estimates before preference collection? This would provide stronger evidence that the observed preference shifts truly arise from belief changes rather than other framing effects.
soundness: 2
strengths: 1. The paper investigates an important, under-explored problem. RLHF typically assumes preferences reflect an (implicit) optimality/regret model. Modeling how beliefs about agent capability influence preferences is a realistic and consequential gap—especially for safety-critical settings where an over-confident preference can induce risky policies. This framing is timely and relevant to the community.
2. The paper provides theoretical guarantee tying disagreement to expected return. Theorem 4.3 and Corollary 4.4 provide an explicit bound showing how the magnitude of agent-labeler disagreement degrades the learned policy’s return. Having a provable dependence is valuable: it provides intuition and a diagnostic for practitioners.
summary: This paper proposes a new theoretical and empirical framework for understanding human beliefs in RLHF. Standard RLHF models assume that human preferences reflect evaluations of trajectories under an optimal agent, but the authors argue that human labelers’ beliefs about agent capabilities also influence their feedback. The paper makes two main contributions. It empirically demonstrates, via synthetic and human studies, that humans’ beliefs about agent competence significantly affect their preferences. Then it formalizes a new “belief-based preference model,” defining an ideal belief alignment between human labelers and agents, and proves theoretical bounds showing how mismatched beliefs can degrade policy performance.

Empirically, the authors show in a GridWorld experiment that preference mismatches between human beliefs and actual agent capability reduce performance. The authors also conduct a human study in a self-driving car setting, showing that priming participants to believe the agent is more or less capable systematically shifts their preferences. The paper concludes by recommending practical strategies for aligning human beliefs and agent capabilities during preference collection—such as transparent communication about agent limitations or online priming during RLHF.
weaknesses: 1. The paper makes strong simplifications in the theory, which limit practical applicability. The theoretical bound (Theorem 4.3) assumes noiseless preferences, deterministic tabular policies, single-transition preference pairs, and that RLHF yields a deterministic policy that exactly respects all preferences. These assumptions are overly idealistic and far from the modern practice of RLHF (noisy annotators, function approximation, and finite data). The practical implication of the bound in realistic RLHF with high-dimensional policies is therefore unclear.
2. The scope of empirical evaluation is narrow. The synthetic experiments are helpful, but they use a toy GridWorld and a specific noise model ($\epsilon$-greedy). It’s unclear how results extend to richer environments (continuous control, long horizon, LLM behavior) or to non-$\epsilon$ noise (e.g., systematic biases, function approximation capacity limits). Besides, the human study is restricted to a single domain (driving) and a particular priming manipulation. While statistically significant effects were found, it’s unclear how general these priming effects are across labeling populations, tasks, and phrasing of instructions.
3. The paper provides no experiments connecting to modern RLHF for LLMs. The paper motivates with general RLHF but stops short of showing how belief-based preferences affect language models or large policy classes used in practice (e.g., direct preference optimization or reward modeling + RL on LMs). The step from gridworld/CARLA to LLMs is nontrivial; without at least a small LM experiment, the generality of modeling human beliefs could be questionable.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Are there any real-world RLHF datasets or collection process that suffer significantly from this belief disagreement problem? It seems to be that if the majority beliefs are "right", then the disagreement for the minority can be viewed as noise and wouldn't make much impact on the learned policy? The synthetic human study confirms that it is possible to manipulate human's preference, but the question still remains on whether it happens a lot in the real world.

2. How should we interpret Theorem 4.3? Can we use the bound for estimating anything, such as how significant would the return be degraded in a given dataset? Also, Theorem 4.3 gives a lower bound on the performance with the mismatched belief, can the authors give an intuition or proof on the tightness of this bound?
soundness: 2
strengths: * The paper is mostly well-written, clearly organized, and easy to follow. It formalizes the agent-labeler agreement clearly, which, based on my knowledge, hasn't been discussed much in previous literature.

* Human study is conducted as evidence that preferences can be shifted in a statistically significant way, which confirms the motivation of this paper.
summary: This paper argues that human beliefs about an agent's capabilities impact their preferences used for RLHF. This mismatch between human's belief and the agent's actual performance would hurt the learned policy. The paper introduces a belief-based preference model and gives theoretical bounds on the performance degradation for the learned policy under this mismatch. Human study is also conducted to verify this claim. The paper finally suggests two approaches that can be applied in actual preference collection process.
weaknesses: * The recommendations given by the authors in Sec 6.1 doesn't seem to be actionable to me (e.g., inform labelers of the known limitation). It still leaves out the problem of *how* or *what* to disclose. As the authors have demonstrated through their human study experiment, these choices can impact the preference in a significant way.

* Section 4 gives us a theory for the effect of human-labeler disagreement, but the authors haven't shown how this theory can be applied for designing any operational protocol. Parameters like $\delta$ in Theorem 4.3 seem to be hard to estimate in real-world scenarios.

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 1
questions: - The authors should clarify the actual assumptions of the model in terms of the actual “knowledge” that should be provided to the labelers.
- Can the authors provide evidence that this is a real problem in practical situations? Can you provide some examples?
- How can you claim that the proposed problem could be generalised to other scenarios?
soundness: 1
strengths: + A formalization of RHLF is per se a very interesting idea.
summary: The authors explore two key questions: whether human labelers’ beliefs affect the preferences they give, and what beliefs would be ideal for optimal learning. They introduce a new model that incorporates these beliefs and provide theoretical bounds showing how mismatched beliefs can harm learning performance. The study is based on human and synthetic experiments.
weaknesses: - A core assumption of this work is that the best possible policy post RLHF is known a priori. But if this is possible, what is the need for RLHF in the first place? At the point, RLHF appears not to be needed? In fact, the authors say that the “normative ideal in regret-based RLHF is for labellers to provide preferences with respect to a quantity that is related to the best possible post-RLHF policy under any labelling”. This appears to be impossible in practice.
- The reviewer understands that it is always nice to see something proven experimentally, but the reviewer would say that it is expected that if there is “disagreement” between the labeler and the agent’s capabilities, the actual performance would be poor.
- In general, the reviewer is not completely sure about the actual overall assumption that is made by the authors: essentially, it seems to me that the authors assume that the insufficient or deceptive information is given to the labelers. How is this something that happens in practice? It seems to me a problem of poor engineering practice that can be tackled with thorough design of the RLHF process.
- Again I understand that there is a potentially theoretical interest in this problem, but it seems to me that in actual fact RLHF is a very practical problem. The solution offered to the practitioners appears not completely viable in my opinion.
- The case study (and the human study) that is presented in the paper is very specific. The authors make some claims about lesson learnt and generalisation that are not really supported by strong evidence in my opinion.
- The formal bound that is presented in the paper refers to a very specific situation. The result is correct, but the reviewer wonders about its practical situation. It is difficult to “approximate” this result to practical situations in my opinion.
- There is an aspect related to the complexity of the task (if you want related to the problem of decision-making in a bounded rationality setting), which is not really taken into consideration by the authors.


Decision: Reject