When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback
Authors: Leon Lang
Davis Foote
Stuart Russell
Anca Dragan
Erik Jenner
Scott Emmons
Venue: NeurIPS 2024 poster (2024)
Keywords: RLHF
Partial Observability
Deception
AI Alignment
Reward Learning
Primary area: safety_in_machine_learning
TL;DR: We study the challenges that arise when learning reward functions with human feedback from partial observations

Abstract: Past analyses of reinforcement learning from human feedback (RLHF) assume that the human evaluators fully observe the environment. What happens when human feedback is based only on partial observations? We formally define two failure cases: deceptive inflation and overjustification. Modeling the human as Boltzmann-rational w.r.t. a belief over trajectories, we prove conditions under which RLHF is guaranteed to result in policies that deceptively inflate their performance, overjustify their behavior to make an impression, or both. Under the new assumption that the human's partial observability is known and accounted for, we then analyze how much information the feedback process provides about the return function. We show that sometimes, the human's feedback determines the return function uniquely up to an additive constant, but in other realistic cases, there is irreducible ambiguity. We propose exploratory research directions to help tackle these challenges and experimentally validate both the theoretical concerns and potential mitigations, and caution against blindly applying RLHF in partially observable settings.

When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback
Leon Lang"
Davis Foote" UC Berkeley
Stuart Russell UC Berkeley
University of Amsterdam
Anca Dragan UC Berkeley
Erik Jenner UC Berkeley
Scott Emmons UC Berkeley
Abstract
Past analyses of reinforcement learning from human feedback (RLHF) assume that the human evaluators fully observe the environment. What happens when human feedback is based only on partial observations? We formally define two failure cases: deceptive inflation and overjustification. Modeling the human Boltzmann- rational w.r.t. a belief over trajectories, we prove conditions under which RLHF is guaranteed result policies that deceptively inflate their performance, overjustify their behavior to make an impression, or both. Under the new assumption that the human's partial observability is known and accounted for, we then analyze how much information the feedback process provides about the return function. We show that sometimes, the human's 's feedback determines the return function uniquely up to an additive constant, but in other realistic cases, there is irreducible ambiguity. We propose exploratory research directions to help tackle these challenges and experimentally validate both the theoretical concerns and potential mitigations, and caution against blindly applying RLHF partially observable settings.
1 Introduction
Reinforcement learning from human feedback (RLHF) and its variants are widely used for finetuning foundation models, including ChatGPT [OpenAI, 2022], Bard [Manyika, 2023], Gemini [Gem- ini Team, 2023]. Llama [Touvron et al., 2023], and Claude [Bai et al., 2022, Anthropic, 2023a,b]. Prior theoretical analysis of RLHF assumes that the human fully observes the state of the world [Skalse et al., 2023]. Under this assumption, from seltzmann--attinna human feedback (see Proposition 3.1).
possible to recover the ground-truth return function
In reality, however, this assumption false. Models like ChatGPT are interacting with the internet and software tools via plugins [OpenAl, 2023]. Software assistants like Devin are interacting with complex IDEs to produce their results [Wu, 2024]. By default, some of the models' work then happens in the background, not observed by the users; see Figure 1. With the tasks performed by language model assistants becoming more complex, is also increasingly time consuming for humans to evaluate the entire model behavior and input. Therefore, we are anticipating a future where by default, the human evaluators do not fully observe environment state that the language assistant is embedded in. Our work analyzes the consequences and risks of such partial observability.
We begin our investigation with simple example, illustrated in Figure 2, meant isolate the key factor leading to deception (in practice, we imagine that this effect would be embedded in a larger,
Core research contributor. Correspondence langouva.......... nl, emmons@erreeley.edd
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

how this year's oscar to recertly released movies for box office sales?
Feedback ck being collected ced
briweing
Orcal vineery office compared recenily rilooved
Reading
that can't see everything the agent sous.
Reading content
The 2023 Oscar alnnar Best Picture "Ewerything Everyatare All Once," earnd $1100
Figure I: Partial observability in ChatGPT [OpenAI, 2023]. Users do observe the online content
that ChatGPT observes yet still provide thumbs-up thumbs-down feedback. OpenAl's privacy policy [OpenAI, 2024c] allows user feedback to be used for training models. We show in Theorem 4.5 that if feedback of human evaluators is based on partial observations, then this can lead to deceptive and overjustifying behavior by the language model.
more complex system, c.g. with logs containing thousands of lines). An Al assistant is helping a user install software. The assistant can hide error messages by redirecting them /dev/nul.. We model the human as having a belief B over the state and extend the Boltzmann goltzmann-rational assumption from prior work to incorporate this belief. In the absence of an error message, the human is uncertain if the agent left the system untouched or hid the error message from a failed installation. If the human interprets trajectories without eтror messages optimistically, the Al learns hide error messages. Figure 4 provides further details on how this failure occurs. also shows a second case where the Al clutters the output with overly verbose logs.
Generalizing from these examples, we formalize dual risks: deceptive inflation and overjustificution We provide mathematical definition of each. When the observation kernel (the function specifying the observations given states) is deterministic, Theorem 4.5 analyzes properties of suboptimal policies learned by RLHF. These policies exhibit deceptive inflation, appearing to produce higher reward than they actually do; overjustification, incurring a cost in order to make good appearance; or both.
After seeing how standard RLHF fails, we ask: What would happen if we would model the human's partial observability correctly in RLHF? Assuming the human's 's belief is known, we mathemati- cally analyze how much information the feedback process provides about the return function. In Theorem 5.2, we show that the human's feedback determines the return function up to a constant and a linear subspace we call the ambiguity. In general the ambiguity may be large enough to allow for arbitrarily high regret, but in some situations the ambiguity vanishes. In experiments that serve proof of concept, we show that explicitly modeling the human's 's partial observability can improve performance, and we offer optimism the form of robustness result (Theorem 5.4) while accounting for the major conceptual difficulties involved. We propose exploratory research directions to solve these issues and improve RLHF situations of partial observability.
2 Related work
The problem of human interpretations of observations was briefly mentioned in Amodei et al. [2017], where evaluators misinterpreted the movement of a robot hand in simulation. Eliciting Latent Knowledge [Christiano et al., 2021] posits that for giving accurate feedback from partial observations, the human needs to be able to query latent knowledge of the Al system about the state. How to do this currently an unsolved problem [Christiano and Xu, 2022]. Recent work [Denison et al., 2024, Wen et al., 2024] provides detailed empirical evidence for deceptive behavior - in line with our notion of deceptive inflation emerging from RLHF based on partial observations, or human evaluators with limited time. The OpenAl ol system card [OpenAI, 2024a] shows that ol sometimes knowingly provides incorrect information or omits important information. Compared to these investigations, and in addition to providing some empirical evidence, we formalize model of human feedback under partial observability, we prove the emergence of failure modes resulting from partial observations, and we investigate potential mitigations.

---

States, actions (unobserved)
Observations
Beliefs
apt install mvidia-driver
Installed nvidia-driver
- +1
apt instoll cuda
dpin:: weror processing cuda
-- -5
exit
O'
apt install nvidia-driver
Installed nvidia-driver
+1
apt Install cuda 2> /dee/mull
+
Suppresses
7
8
π install cuda
>> /dev/nut1
Figure 2: A human compares trajectories to provide data for RLHF. Rather than observing and s", the human observations ő and o", which they to estimate the total reward of each trajectory. In this intentionally simple example, an agent executes shell commands to install Nvidia drivers and CUDA. Both 5 and 5" contain error, but in s", the agent hides the error. The human believes s" better than 8, rewarding the agent's deceptive behavior. The underlying MDP and observation function are in Figure 7.
Our work argues that deception result from applying RLHF from partial observations. Deception may also emerge for other reasons: Hubinger et al. [2019] introduced the hypothetical scenario of deceptive alignment, in which an AI system deceives humans into believing it aligned while it plans later takeover. Under the definition from Park et al. [2024b], GPT-4 was shown to behave deceptively in a simulated environment [Scheurer al., 2023]. A third line of research defines deception in structural causal games and adds the aspect of intentionality [Ward et al., 2023], with recent preliminary empirical support [Hofstätter al., 2023]. We outline more related work in Appendix B.
3 Reward identifiability from full observations
Here we review Markov decision processes and previous results on reward identifiability under RLHE
3.1 Markov decision processes
We assume Markov decision processes (MDPs) given by (S,A, 7. For any finite X, let A(X) be the set of probability distributions on X. Then S is finite set of states, A is a finite set of actions, T: S A (SS) is a transition kernel written T(s' s, s,a) [0, 1). Po A(S) is an initial state distribution, R S + R is the true reward function, and E [0, 1] is a discount factor.
A policy is given by function π S A(A). We assume a finite time horizon T.. Let Š be the set of possible state sequences 5 = .o...... .T. so & E & if it has strictly positive probability of being sampled from Po. T, and an exploration policy т with πt 8) > for all E S,a E A. A sequence & gives rise to return G( = Σ 7 R(s)). ). Let pa (5) be the on on-policy probability that is sampled from Po. T, æ. The policy then usually trained to maximize the policy evaluation function J, which is the on-policy expectation of the return function: ((T) = Eg-Px() [G(3)]
3.2 RLHF and identifiability from full observations
In practice, the reward function R may not be known and need be learned from human feedback. In simple form of RLHF [Christiano et al., 2017], this feedback takes the form of binary trajectory comparisons: a human presented with state sequences š and s" and choose the one they prefer. Under the Boltzmann rationality model, we assume the human picks & with probability
pa p7(6z = (
(1)
3

---

where B > O is an inverse temperature parameter and (x)) : 1+exp(-2) is the sigmoid fune- tion [Bradley and Terry, 1952, Christiano et al., 2017, Jeon et al., 2020].
1
An important question is identifaaiiiity In the infinite data limit, do the human choice probabilities PR collectively provide enough information to uniquely identify the reward function R? This is answered by Skalse et al. [2023, Theorem 3.9 and Lemma B.3]:
Proposition 3.1 (Skalse et al. (2023]). Let R be the true reward function and G the corresponding return function. Then the collection of all choice probabilities (R (8 > 8") for state sequence pairs 3,8" E S determines the return function G on sequences 5 E S up to an additive constant.
The reason is simple: because a is bijective, P determines the difference in returns between any two trajectories. From that
reconstruct individual returns up an additive constant.
The reward function R is not necessarily identifiable from preference comparisons; see Skalse et al. [2023, Lemma B.3] for precise characterization. However, the optimal policy only depends on R indirectly through the return function G, and invariant under adding a constant to G. Thus in the fully observable setting, Boltzmann rational comparisons completely determine the optimal policy. In Section 5, we show conditions under which this guarantee breaks in the partially observable setting.
The impact of partial observations on RLHF
We now analyze failure modes of a naive application of RLHF from partial observations. both theoretically and with examples. In Proposition 4.1, we show that under partial observations, RLHF incentives policies that maximize what we call Johs policy evaluation function that evaluates how good the state sequences "look to the human". The resulting policies can show two distinct failure modes that we formally define and call deceptive inflation and overjustification. In Theorem 4.5 we prove that at least one of them is present for Jobs Jobs-maximiize maximizing policies. Later, in Section 5, we will see that adaptation of the usual RLHF process might sometimes be able avoid these problems.
To model partial observability, we introduce an observation space E Z and observation kernel with probabilities Po 8) E [0, 1]. We write Po(ō 8) == II Po 8x) for the probability of an observation sequence. We write Ω for the set of observation sequences that occur with non-zero probability, i.e., ō Φ if and only if there is 5 & such that I Po(o: 8t) > 0. If Po and Po deterministic, then we write O : S > Ω and Ō 5 + Ō for the corresponding observation functions with (s) and (3) ō for o and ō with Po s) = and Pā(ō 8) = 1, respectively.
4.1 What does RLHF learn from partial observations?
We consider the setting where the state is fully observable to the learned policy, but human feedback depends only sequence of observations. We assume that the human gives feedback under Boltz- mann rational model similar to Eq. (1), modified such that they form some belief ( ō) E [o, 1] about the state sequence 5 based on the observations ō. We then assume preferences are Boltzmann rational in the expected returns under this belief, instead of the actual returns.
The assumption of Boltzmann rationality is false in practice [Evans et al., 2015, Majumdar et al., 2017, Buehler et al., 1994], but note that it is an optimistic assumption: Even though our model is simplification, expect that practical issues can be at least bad as the ones we will discuss. See also Example E.4 for an example showing that it is sometimes generally not possible to find human model that leads to good outcomes under RLHF. Future work could investigate different human models and their impact under partial observability in greater detail.
To formalize our setting, we collect human beliefs into a matrix B = (B(3) 5)) 0. E R The
expected returns for observations ő are given by Eg- [G(5)] (B (O). We view G E R
and B G R as both column vectors and functions. Plugging these expected returns into Eq. (1) gives
p" oo = =
(B-G)('))) (
(2)
This is an instance of reward-rational implicit choice [Jeon et al., 2020], with the function ō - B( )
as the grounding function. If observations are deterministic, write Õ(3) ō for ø with

---

Po(ō1 8) 1. We can then recover the fully observable case Eq. (1) with B and Ô being the identity.
The belief B can be any distribution as long as it sums to 1 over 8. The human could arrive at such belief via Bayesian updates, assuming knowledge of Po. T, Po. and prior over the policy that generates the trajectories (see Appendix D.I). None of our results rely on this more detailed model.
We assume the human gives feedback according Eq. (2) but the system uses the standard RLHF algorithm based on Eq. (1), We define the following observation return function Gober and we show in Appendix E.1 that if observations deterministic, RLHF infers this up to additive constant.
[ (B)))
Gobs
E
For deterministic Pg. this can be simplified to Goha ( (O(5) where 5)) =1 Note that deterministic observations can be ambiguous multiple states produce the same observation.
Unlike in the fully observable case of Proposition 3.1, a return function might be inferred that implies an incorrect of optimal policies. We define the resulting policy evaluation function Jobs by
(a)
(5))
P~
This is the function which standard reinforcement learning algorithm would optimize given the inferred return function Gobs- We summarize this as follows:
Proposition 4.1. In partially observable settings with deterministic observations, a policy is optimal according RLHE, i.e., according return function model that would be learned by RLHF with infinite comparison data, if it maximizes Jobs-
Note that in this definition, and specifically in the formula for Gobss the human does not have knowledge of the policy ı that generates the state sequence n. In Appendix E.2, we briefly discuss the unrealistic case that the human does know the precise policy and is an ideal Bayesian reasoner over the true environment dynamics. In that case, Johs = J, i.e. there is no discrepancy between true and inferred returns. Intuitively, even if the human would not make any observations, they could give correct feedback essentially by estimating the policy's expected return explicitly.
In case, however, a policy achieving high Jobs produces state sequences 5' whose observation sequence Ö(5) looks good according to the human's belief (5" Ö(5)). This hints a possible source of deception: if the policy achieves sequences whose observations look good at the expense of actual value G(5), we might intuitively call this deceptive behavior. We now analyze this point in greater detail.
4.2 An ontology of behaviors
We will evaluate state sequences based on the extent to which they lead to the human overestimating or underestimating the reward in expectation. Recall that Gobs from Equation (3) measures the expected return from the perspective of human with some belief function B and access to only observations, whereas G are the true returns. That leads us to the following definition:
Definition 4.2 (Overestimation and Underestimation Error). Let be state sequence. We define its overestimation error E and underestimation error E by
F+ (8) == max (0, Gobs (8) G(
E (3) max (0, G(5) Gobs
We further define the average overestimation (underestimation) error under a policy by E (TT) = E. (F+) and E (m) = Eg EE (5)).
We consider policy π in comparison to some reference policy Rea This can loosely be understood as counterfactual policy in the absence of some intervention, where и the factual policy resulting from the intervention. We discuss increases and decreases in over- and underestimation error which are implicitly due to some intervention. will be the Jobs-optingg optimal policy; the "intervention" is thus the introduction of partial observability.
our purposes, ner will be the true optimal policy, and
5

---

Figure 3 shows simple ontology of behaviors that increase and decrease the average over- and underestimation error. Increasing either of these quantities decreases the accuracy of the human'ss estimates, and can thus be thought of as "misleading", decreasing either of them improves accuracy and can be thought of "informing".
4.3 Deceptive inflation and overjustification
Standard RLHF in the setting of partial observa- tions incentivizes undesirable forms inflating and justifying. We refer to the philosophical definition of deception offered by Park et al. [2024b],
Underestination error
Overestination arror
E
E
#icalignes KLA Incencivvises Chase
Increase
"the systematic inducement of false be- liefs the pursuit of some outcome other than the truth." "
Misleading
Downplaying Causing mis Lakknny pessieiatie estipates.
Inflating Causing optintstbl est estisates.
to anchor the notion that increasing the over- estimation eтrоr in order to improve the RLHF objective Jabs is deceptive, leading to the fol- lowing definition.
torr Infominng
Justifying Correcting ing edstakeniy posss fsiitll
Disillusioning
Definition 4.3 (Deceptive Inflation). A policy x exhibits deceptive inflation relative to we if
Correcting inistalionyy apElelstic
E
(TT F. (mrr) and Johs (T) > (Tn))
We typically prefer that our Al agents engage in informing behaviors. Undesirable inform- ing behaviors decrease reward despite provid- ing information. We name undesirable justify- ing behaviors "overjustification" as nod to the overjustification effect from psychology [Deci and Flaste, 1995], in which subjects become de- pendent on an extrinsic source of motivation to sustain work task.
(eccrese
Figure 3: Behaviors defined by increasing and de- creasing the human's 'ss over- and underestimation error. RLHF with partial observations results in incentives to increase overestimation error and de- crease underestimation error (Theorem 4.5),
Definition 4.4 (Overjustification)) A policy exhibits overjustification relative to Rieef if E (T) < E (Trz)) and (T) < J(Tr)))
To understand the counterintuitive notion that an agent providing information to the human could be undesirable, consider a PhD student who looks to feedback from their advisor for direction. They meet for one hour a week. Suppose the student explain last week's work in 15 minutes, leaving the remaining time to discuss next steps. They could instead "overjustify" by spending the entire hour going through the last week's work in far more detail, leaving no time for next steps. From the advisor's perspective, the latter is more informative, but is worse allocation of limited resources.
We now state a key result. See Appendix E.3 for the proof.
Theorem 4.5. Assume that Po is deterministic. Let IT be the set of optimal policies according to a naive application of RLHF under partial observability, and les II be the set of optimal policies according to the true objective J. If xt E TT IT' and The E TI* least one of deceptive inflation or overjustification relative to π.
obss II,, then The must exhibit at
Note that trajectory may be more or less likely under &h than '² regardless of human estimation. so long as on net Thes exhibits deceptive inflation or overjustification.
Our analysis extends beyond the special case of RLHF to inverse preference learning (IPL) [Hejna and Sadigh, 2023], and thus to direct preference optimization (DPO) [Rafailov al., 2023], which IPL generalizes. Theorem in Hejna and Sadigh [2023] shows that IPL will converge to policy that maximizes an implicit reward function that matches the human's 's preference judgments as well as possible. If the human's 's preference judgments come from partial observations, then the resulting return function will be Gabs, as we describe in our discussion leading up to Proposition 4. 1. This leads to the same problems of deceptive inflation and overjustification that describe in Theorem 4.5.

---

Observations
Beliofs
Failure mode
1
Apest didnt cry
PP <
(Ia krman optinetti oF IAADON
A. Deceptive Inflation
Instalted midio driver
r=0
Pick
Sapal seat dee B/see/nal craet
better feechack
celenklri
las Dudles/mal: that
Utanbigunes atservation
Iastalled weldia-deivet Installatton fatted
p=1: Ale Seltel meral Oge
Wigigs obserration
Ane Jndelled Vayly
Mpune
li proisaation sleptioa)
Installed Pyshon
B. Overjustification
Agent dide'c acceupt install
pee better Inseal when deec must de onding???
Unenbigueus (verbese)
Checking dependencies....
rucatied suily. Hoppre
P-
Figure 4: Scenarios illustrating failure modes due partial observability. In each, the agent must install two packages. Formal details of the underlying MDPs are provided in Appendix C. A, top: In the absence of log message about CUDA, the human unsure whether the agent skipped it or used the 2> /dev/null trick (see Figure 2): if the human is insufficiently skeptical, the trick looks optimal to the agent. B, bottom: Default logging in this case is silent when the NumPy install is successful. The agent can optionally use a --verbose flag, but this produces long log that the human prefers not to see. If the human is too skeptical, verbose logging still appears optimal to the agent.
4.4 Deception and overjustification in examples
We now theoretically analyze intentionally simple examples that illustrate the failure modes of RLHF in the presence of partial observability. Our examples feature an agent trained with RLHF to complete tasks in a user's terminal. The output of each command is piped to a log file, which is what the human evaluator sees when making choices for RLHF. For mathematical details about these examples, including a tutorial-style analysis of Example A with derivations, see Appendix C. For an empirical validation of these theoretical concerns and proof of concept on how to mitigate the issues, see Section 5.2.
In Fig. 4A, the agent's task is to install Nvidia drivers and CUDA. The agent may skip the CUDA installation, or may attempt it with or without 2> /dev/null (which suppresses errors) appended. If the log contains no message about CUDA, the human believes with probability Paide that the agent used the 2> /dev/null trick and the install failed, and with probability - Plide that the agent didn't 't attempt the CUDA install. If Paide is sufficiently small, then RLHF will learn from the resulting feedback that the optimal policy is to use the 2> /dev/null /null trick. This is deceptive inflation, because this policy produces trajectories for which the human systematically overestimates the return.
In Fig. 4B, the agent's task is to install Python and NumPy. The agent can skip the NumPy install, or attempt it with either "default" settings that only logs errors or with verbeee flag that adds additional info. In the case of a success, the human prefers not to see this verbose output. log contains no message about NumPy, the human believes with probability Paefmult that the agent attempted the install with default logging and succeeded, and with probability 1 1-Pdetautt that the agent skipped the NumPy install. If Pdefautt is sufficiently small, then RLHF will learn from the resulting feedback that the optimal policy is to use - veerbose logging. This is overjustification, because the agent chooses a suboptimal policy in order to avoid the human underestimating its performance.
the
5 Return ambiguity from feedback under known partial observability
We've seen issues with standard RLHF applied to feedback from partial observations. Part of the problem is model misspecification: the standard RLHF model implicitly assumes full observability. Assuming the human's 's partial observability is known, could one do better?
We start Section 5.1 by analyzing how much information the feedback process provides about the return function when the human's 's choice model under partial observations is known precisely. We show that the feedback determines the correct return function up to an additive constant and linear subspace we call the ambiguity (Theorem 5.2). If the human had return function that differed from the true return function by an element in the ambiguity, they would give the exact same feedback such return functions are thus feedback-compatibe We then show an example where the ambiguity
7

---

Reward functions (on states)
Return functions (on state sequences)
B
E[Return] fnnttrronfnnniions functions (on obs sequences)
rnviromenn dynanics
R
F-
Ctane separaaee) ream furctiors
im r n ker
B
Adding these EREan's Loder ln Innae's bellef
R
R'
labiguity
0
R+R" R
G+G
G
BG
Recam faactices CEAN retem C
G?
G+G?
Figure 5: By Theorem 5.2, even with infinite comparison data and access to the correct human model, a hypothetical reward learning system (depicted as a robot) could only infer G up to the ambiguity im ImrKken B (purple). Adding an element of the ambiguity to G leads to the exact same choice probabilities for all possible comparisons, and the reward learning system has no way to identify G' among the return functions in G + (im I' ker B) (yellow). This abstract depiction ignores the linearity of these spaces; for a more precise geometric depiction of B, see Figure 8 in the appendix.
vanishes, and another where it doesn't leading to (eedback-compatibl) return functions that have optimal policies with high regret under the true return function. Finally, in Section 5.2 we explore Theorem 5.2 starting point design reward learning techniques
how could in theory that work under partial observability. In particular, we experimentally show in proof of concept that being aware of the human's 's partial observability improves performance. this section do assume Po to be deterministic.
5.1 Feedback-compatiiblig and ambiguity of return functions
Assume that the human gives feedback based on the choice-probabilities from Eq. (2). the infinite data limit, can be assumed that the whole collection of probabilities
is known since the choice frequencies approach these probabilities. Here, we write PG instead of PR since the reward function only enters the choice probabilities through the corresponding return function G. The question answer in this section is how much information the choice probabilities provide about G, assuming the human choice model known and correct. The choice probabilities tell us precisely that the true return function gives rise to these choice probabilities, i.e., is feedback-compatible. This captured the following definition:
5,
Definition 5.1. Let (po (0>)')) 5,5'
be the vector of choice probabilities and G return function
corresponding to reward function R. Then G is feedback-compattbn (with respect the vector of choice probabilities) if (ā o') pG (ō for all ö, o' 52.
Crucially, without further assumptions inductive biases, no learning algorithm can pick out the true return function among beedback-compatible return functions. thus crucial to know whether return functions that are unsafe when using them to optimize a policy.
there are
We now determine the set of feedback-compatibl return functions. Write I Rx5 for the matrix that maps reward function its return function, i.c. (R) 2 R(((s.. Its matrix elements are given by r F. Et= (se)),,, where (st) = 1(s }.). Then the image im I is the set of all return functions that can be realized from reward function given the MDP dynamics T. Recall the belief matrix B- = (B(5)) σ)) प. E R Taking into account that itself is im I and )))=.z
that G enters the choice probabilities only through B G meaning that the choice probabilities do not vary if we change additively up an element in the kernel ker we obtain the following result:

---

Theorem 5.2. Let the collection of choice probabilities be given by (Pm (p"(ō>0') o')
a, following if and
a Boltzmann rational model as in Eq. (2). Then a return function G is
only if there is G' ker B im I and с R such that G = G + G' + c. In particular, the choice probabilities determine G up to an additive constant if and only if ker B 3 = {0}.
See Theorem D.2 and Corollary D.4 for full proofs, and Figure 5 for a visual depiction. This result motivates the following definition:
Definition 5.3 (Ambiguity). We call ker B 8iimI I the ambiguity that is left in the return function when the human choice model and observation- -based choice probabilities are known.
How large the return ambiguity? For Fig. 4A, one can show that the ambiguity is nontrivial, allowing for
return functions with unsafe optimal policies. Intuitively, since successfully installing CUDA produces the same observation regardless of whether 2> / /dev/null /null was used, the choice probabilities don't 't give any information to determine distinct reward values for these two outcomes, only their average over the human's 's belief upon observing a successful install. Thus, reward functions assigning arbitrarily high reward to success with 2> /dev/null are bredback-compatiin
Such reward functions can then lead to an incentive for learned policy to hide the error messages even with a correct observation model. More details can be found in Appendix C.4.
We saw in Fig. 4B case where naive RLHF under partial observability can lead to overjustification. However, the human's 's feedback and belief model actually provide enough information to determine the return function. The reason is that ker B leaves only one degree of freedom that is not "time- separable" over states, and thus ker B n nim r {0]. More details can be found in Appendix C.4.
5.2 Toward improving RLHF in partially observable settings
To improve RLHF when partial observability is unavoidable, one could take Theorem 5.2 as starting point to find a learning algorithm that converges to feedback-compatible return functions. This would require the human model to be fully known and specified, including knowledge of the belief probabilities B(x1 | ō), which can differ from human to human. If one assumes the human is rational, as Appendix D.I, this requires specifying the human's policy prior (TT).. Instead of directly specifying these models, one could also attempt to learn a generative model for B ō). These problems reveal further conceptual challenge: for complex environments, humans do not form beliefs over the entire environment state s. A better starting point for practical work may thus be to model humans as forming expectations over reward-relevant features of the state.
If B were explicitly known, one could in principle encode B into the loss function of an adapted RLHF process to learn concept, we used this procedure to analyze the examples in Figure 4 empirically, see Table 1. We do this by first learning a reward model by logistic regression against the true choice probabilities of synthetic human under partial observability. and then learning the optimal Q-function of the resulting reward model with value iteration. The resulting policy chooses a unique action after installation of the nvidia driver (Example A) or Python (Example B) as listed in the "action" column.
return funetion; see Appendix D.3. As a proof of
Table 1 shows that in 3 of four cases, being "partial observability aware" ("po-aware") leads to the true optimal policy when "naive" RLHF does not. In the one case where being "po-aware" does not improve performance (second line in the table), this is explained by the fact that there is remaining ambiguity in the return function. Curiously, in line 4 our theory also predicts remaining ambiguity, but the optimal policy is learned; we consider this to be luck. We provide more details on our experiments in Appendix C.5.
As already demonstrated, tredback-compattee ambiguity. In Example D.29, we even show a case where some feedback-compp have optimal policies that are even worse than simply maximizing Jabs An important direction for future work is to investigate learning algorithms and inductive biases that help "find" safe return functions among all those that are uncertainty. Another line of inquiry is to determine when the set of coodback-compation return functions is "safe", which depends on the MDP, observation function, and human model.
return functions can be unsafe due to remaining
return functions
that act conservatively given the
One sufficient condition for feedhack-compatibn return functions to safe is the vanishing of the ambiguity ker B n im I. Even then, one realistically still has to deal with the problem that B is at

---

Table 1: Experiments showing improved performance of po-aware RLHF
Ex.
p
Phide
Pdefault
model
action
dec. infl.
E
overj.
optimal
A
0.5
0.5
N/A
naive
6n
1.5
0
X
X
0.5
0.5
N/A po-aware
a
1.5
x
X
A
0.1
0.9
N/A
naive
aC
x
0
V
X
A
0.1
0.9
N/A
po-aware
0r
0
5.4
x
V
B
0.5
N/A
0.9
naive
4T
4.5
V
x
B
0.5
N/A
0.9
po-aware an
0
0.25
X
V
0.5
N/A
0.1
naive Qv
B
0.5
N/A
0.1
po-aware
a
0
x
2.25
X
best known approximately. Fortunately, in Appendix D.6, we prove that small errors in the assumed belief matrix lead to only small errors in the inferred return function:
Theorem 5.4. Assume ker B 3nim r = {0}. Let B = B - A be small perturbation of B, where A || S p for sufficiently small p. Let be the true return function and assume that a hypothetical learning system, assuming the human's belief is Ba, infers the return function G with the property that B -G has the smallest possible Euclidean distance to B -G.
Let rBB) = B lmr be the (injective) restriction of the operator B to im Then r(B)rr(B) is invertible, and there exists a polynomial Q(X, Y) of degree 5 such that
G Gll S ||G|| -lll-oQ (I((((())T "II
In particular, show in the appendix, can uniformly bound the difference between Ja and Jg. This yields a regret bound between the policy optimal under G and an optimal policy x² for G.
There are also alternatives to modeling the human belief B. For example, one could mix human evaluations based high-cost full observations and low-cost partial observations for finding an optimal tradeoff [Mallen and Belrose, 2024]. Finally, it would help if the human could query the policy about reward-relever aspects of the environment to bring the setting closer to RLHF from full observations. This is similar to the problem of eliciting the latent knowledge of predictor of future observations [Christiano al., 2021, Christiano and Xu, 2022]. While this may avoid the need to specify the human's 's belief model (3 o), it requires understanding and effectively querying an ML model's belief, including translating from an ML model's ontology into a human ontology.
Conclusions
In this paper, we provided a conceptual and theoretical investigation of challenges when applying RLHF from partial observations. First, we saw that applying RLHF naively when assuming full observability can lead to deceptive inflation and overjustification behavior. Then, we showed that even when the human's 's partial observability known, the of feedback compatibble return functions can contain irreducible ambiguity. This means that without further inductive biases, no learning algorithm can generally be expected to infer the correct return function. Finally, we recommended further exploratory research to study and improve RL.FF for cases when partial observability is unavoidable and provided a proof of concept that modeling the human's 's partial observability can improve performance. In conclusion, we recommend caution when using RLHF in situations of partial observability, and hope that further research studies the effects in practice and helps to address these challenges.
Limitations We assume the human be Boltzmann rational and to implicitly compute an expected value of the return, which is unrealistic for actual humans. Other types of choices could be considered, as in seward-rationa choice [Jeon et al., 2020] and assistance games Finally, we assume that the human forms belief B ō) over the true state sequence ø. If the envi- ronment is complex, humans will in reality only form heliefs over lower-dimenssinnll representations or features of the state.
Maddield-Menede et al., 2016).
10

---

Author contributions
The project conceived in parallel by Scott and Davis, with key shift proposed by Leon. Leon proved Proposition 4.1 and Theorems 5.2 and 5.4, found the first mathematical examples of what became deceptive inflation and overjustification that can be resolved by Theorem 5.2, and wrote the majority of the appendix. Davis conjectured Proposition 4.1, provided early empirical evidence that RLHF under partial observations can lead to deception (not the paper), defined deception / deceptive inflation and overjustification (with Scott), proved Theorem 4.5, and developed the running examples and figures. Scott guided the project direction and prioritization, gave the conjecture and proof idea for Theorem 5.4, and helped develop the running examples and deception definitions. Erik provided regular detailed feedback and guidance and edited the paper. Anca and Stuart advised this project.
Acknowledgments and Disclosure of Funding
Leon Lang thanks the Center for Human-Compiin Compatile Artificial Intelligence for hosting him during part of this project, and Open Philanthropy for financial support. All authors thank Open Philamthropy for its support of the Center for CompaiCCoea Artificial Intelligence. Davis was supported by the Berkeley Existential Risk Initiative. Erik was supported by fellowships from the Future of Life Institute and Open Philanthropy. We thank Benjamin Eysenbach and Benjamin Plaut for detailed comments and feedback on this work, and we thank Elio A. Farina, Mary Marinou, and Alexandra Horn for assistance with graphic design.
References
D. Amodei, P. Christiano, and A. Ray. Learning from human preferences. https://openai.coo esearch/leerniin-frree hww.oom-mmmmmppppp hraeraanneeeroauuuuaaaarrrreeeee 2017. Accessed: 2023-12-13.
Anthropic. Introducing Claude.
2023a. Accessed: 2023-09-05.
Anthropic. Claude's Constitution. http:///www..atttpppooooc/////as
tion, 2023b. Accessed: 2023-09-05.
Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirho- seini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, Mueller, Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. Lasenby, R. Larson, S. Ringer, S. Johnston, S. Kravec, El Showk, S. Fort, T. Lanham, T. Telleen-Lawton, T. Conerly, T. Hemighan, T. Hume, S. R. Bowman, Z. Haadield-Dodds, B. Mann. D. Amodei, N. Joseph, S. McCandlish, T. Brown, and J. Kaplan. Constitutional AI: Harmlessness from AI Feedback. arXiv e-prints, art. Stti::2212.00077: Dec. 2022. doi: 10.48550/arXiv.2221..00
R. A. Bradley and M. E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4):324-34, 1952. ISSN 00063444. URL http:
// www. jjttr.str... org/ sttg///aaeee/2342222
R. Buchler, D. Griffin, and M. Ross. Exploring the "Planning Fallacy": Why People Underestimate Their Task Completion Times. Journal of Personality and Social Psychology, 67:366-381, 09 1994. doi: 10. 10.1037/0022-351 -51022255144600 35.......4....66666
C. Burns, H. Ye, Klein, and J. Steinhardt. Discovering Latent Knowledge in Language Models Without Supervision. In The Eleventh International Conference on Learning Representations, 2023. URL //tttp:///ppereveeeee
Casper, X. Davies, C. Shi, T. K. Gilbert, Scheurer, J. Rando, R. Freedman, Korbak, D. Lindner, P. Freire, T. Wang, S. Marks, C.-R. Segerie, M. Carroll, A. Peng, P. Christoffersen, M. Damani, S. Slocum, U. Anwar, A. Siththaranjan, M. Nadeau, E, J. Michaud, Pfau, D. Krasheninnikov, X. Chen, L. Langosco, P. Hase, F. Biyik, A. Dragan, D. Krueger, D. Sadigh, and D. Hadfield- Menell. Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. arxiv e-prints, 2023.
11

---

K. Chidambaram, K. V. Seetharaman, and V. Syrgkanis. Direct Preference Optimization With Unobserved Preference Heterogeneity, 2024. URL /tt.ttsss//ara.v.o///aa//////// 15065.
P. Christiano and M. Xu. ELK prize results. http::/www.aliguuooffft...ommm SB2Xccn9qi5t/elll prize-results, 2022. Accessed: 2024-02-15.
wrg.
P. Christiano. J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep Reinforcement Learning from Human Preferences. arXiv e-prints, art. arXiv:1706.03741, June 2017. doi: 10.44855/arxivv.1776. 170.00334...
Christiano, Cotra, and M. Xu, Eliciting Latent Knowledge, https://docs .oggg/// com/ Document/d// http://dddwwwwn//t/esee Yh- th-ChhXRLaH BwwnaascasascassaPPeoooooo 2023-04-25.
2021. Accessed:
E. L.. Deci and R. Flaste. Why we do what we do: The dynamics of personal autonomy. GP Putnam's Sons, 1995.
C. Denison, M. MacDiarmid, F. Barez, D. Duvenaud, S. Kravec, S. Marks, N. Schiefer, R. Soklaski, A. Tamkin, J. Kaplan, B. Shlegeris, R. Bowman, E. Perez, and E. Hubinger. Sycophancy to Subterfuge: Investigating deward-Tamperiin in Large Language Models, 2024. URL https: // //arxiv.org abs 2406. 10162.
L. El Ghaoui. Inversion emror, condition number, and approximate inverses of uncertain matrices. Linear Algebra and its Applications, 13-334:1171-193 2002. ISSN 0024-3795. doi: https: / http///t...eeep////////5//000100 Linear equations.
URL https: :ttp:://www./// sciencedirect. com/scis Special Issue on Structured and Infinite Systems of
O. Evans, A. Stuhlmueller, and N. D. Goodman. Learning the Preferences of Ignorant, Inconsistent Agents, arxiv e-prints, 2015.
O. Evans, O. Truthful AI: Developing and Governing Al that does not lie. arxiv e-prints, 2021.
L. Finnveden, A. Bales, A. Balwit, P. Wills, L. Righetti, and W. Saunders.
A. Fern, S. Natarajan, K. Judah, and P. Tadepalli. A Decision-Theoretic Model of Assistance. J. Artif. Int. Res., 50(1):71-104, may 2014. ISSN 1076-9757.
D. Geiger, T. Verma, and J. Pearl. Identifying independence in bayesian networks. Networks, 20: 507-534, 1990. URL httpt:ttt//ppp semanticscholas IDgg/orrpsII:::11 1938713.
G. Gemini Team. Gemini: A Family of Highly Capable Multimodal Models. httpss//stons e. googleapis wwt.cooo//eeeeeeeeeeee conn/depmiid-media. gemini/ gemini/gemini..e.. gremini_1_reporr_ teff 2023. Accessed: 2023-12-11.
D. Hadfield-Menell, A. Dragan, Abbeel, and S. Russell. Cooperative Inverse Reinforcement Learning. arXiv e-prints, art. arXiv:1606.03133: June 2016. doi: 00.48550/axiiv.6606.31113
Hejna and D. Sadigh. Inverse Preference Learning: Preference-basse based RL without Reward Function. arXiv e-prints, art. trXiv:2305.15363 May 2023. doi:
Hofstätter, F. R. Ward, HarrietW, L. Thomson, O. P. Bartak, and F. Brown. Tall Tales at Different Scales: Evaluating Scaling Trends for Deception in Language Models. https: //www www.aaentoommnoormmmmmmmm -scales--valuatting scaling- twwntteeedddeeee 2023. Accessed: 2024-01-23.
aasaalllllleleeeeeeeeeeeeeeeleae at different wtsps//s/pppsStEEAEEEEEEEEES
L. Huang, W. Yu. W. Ma, W. Zhong, Z. Feng, H. Wang. Q. Chen, W. Peng. X. Feng, B. Qin, et al. Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. arXiv preprint arXiv:231 000o:23/1.0022.. 1.02232, 2023.
E. Hubinger, C. van Merwijk, V. Mikulik, J. Skalse, and S. Garrabrant, Risks from Learned Optimization in Advanced Machine Learning Systems. arXiv e-prints, art. arXiv:1906.00 001.00000000.0..0000000000000000 June 2019. doi: 10.4 48.4 wwt.xii...96.0111 1906.01820.
12

---

H. J. Jeon, S. Milli, and A. Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lim, editors, Advances in Neural Information Processing Systems, volume 33, pages 4415-4426. Curran Associates, Inc., 2020. URL
C. Kausik, M. Mutti, A. Pacchiano, and A. Tewari. A Theoretical Framework for Partially Observed Reward-States in RLHF, 2024. URL. https: : h////aaxiv wtwpi/..rrg/bb//////0// httpaa////22202/// 03282.
S. Lin, J. Hilton, and Evans. TruthfulQA: Measuring How Models Mimic Human Falsehoods. arxiv e-prints, 2022.
A. Majumdar, S. Singh, A. Mandlekar, and M. Pavone. Risk-sensitive inverse reinforcement learning via coherent risk models. In N. Amato, S. Srinivasa, N. Ayanian, and S. Kuindersma, editors, Robotics, Robotics: Science and Systems, United States, 2017. MIT Press Journals. doi: 10.15607 /rss.2017.xii..69..
A. Mallen and N. Belrose. Balancing Label Quantity and Quality for Scalable Elicitation, 2024. URL https: /ctps://arxiv..e wrg.:///////a//a//eeeee 13211
J. Manyika. An overview of Bard: an early experiment with generative AL atttps://a.. https://ai.gooole pdf, 2023. Accessed: 2023-09-05.
staticc/dccumnts// http://cccummettggggggoooeeoeeee
Mindermann and Armstrong. Occam's Razor is Insufficient to Infer the Preferences of Irrational Agents. In Proceedings of the 32nd International Conference Neural Information Processing Systems, NIPS' 18, page 5603-5614, Red Hook, NY, USA, 2018. Curran Associates Inc.
A. Y. Ng. S. Russell, et al. Algorithms for Inverse Reinforcement Learning. In ICML, volume 1, page 2, 2000.
OpenAI. Introducing ChatGPT. 2024-02-06.
2022. Accessed:
OpenAl. ChatGPT Plugins. httper//.peeaaaaccc///hh////ttts 2023. 2024-05-22,
2023. Accessed:
OpenAI. OpenAl ol System Card, 2024a. pdf. Accessed: 2024-10-28,
OpenAI. Model Spec, 2024b. URL 8. html.. Accessed: 2024-10-28.
OpenAl. Privacy Accessed: 2024-05-22.
https: http://ttooo///eaaaoo//ooo/ttttr 2024c 2024c.
C. Park, M. Liu, Kong, Zhang, and A. Ozdaglar. RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation, 2024a. URL 54.
Park, S. Goldstein, O'Gara, Chen, and D. Hendrycks. Ai deception: A survey of examples, risks, and potential solutions. Patterns, 5(5), 2024b.
R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct Preference Optimization: Your Language Model Secretly Reward Model. arxiv e-prints, 2023.
Scheurer, M. Balesni, and M. Hobbhahn. Technical Report: Large Language Models can Strategi- cally Deceive their Users when Put Under Pressure. arxiv e-prints, 2023.
R. Shah, D. Krasheninnikov, J. Alexander, P. Abbeel, and A. Dragan. The Implicit Preference Information in an Initial State. In International Conference on Learning Representations, 2019. wtt.://www.......errmmee/eee///n URL /ttp
R. Shah, P. Freire, N. Alex, R. Freedman, D. Krasheninnikov, L. Chan, M. D. Dennis, P. Abbeel, A. Dragan, and S. Russell. Benefits of Assistance over Reward Learning, 2021. URI, https:
//penreview www/ att//forumnid-DDLooDDeeii
13

---

A. Siththaranjan, C. Laidlaw. and D. HadfelldMMenll.. Distributional Preference Learning: Un- derstanding and Accounting for Hidden Context RLHF. arXiv preprint arXiv:2312.08358, 2023.
J. Skalse and A. Abate. Misspecification in Inverse Reinforcement Learning. arXiv e-prints, art. arXiv:2212.03201, Dec. 2022. doi: 10-4850VVaXiv:222..3300
J. M. V. Skalse, M. Barrugia-Roberte S. Russell, A. Abate, and A. Gleave. Invariance in Policy Optimisation and Partial Identifiability in Reward Learning. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pages 32033- 32058. PMLR, 23-29 Jul 2023. URL https: httpaa///ppcpedi/ippe .mr..mlr... www.ppres/vvee sreess/2202skaaase/22 .html.
Stray. The AI Learns Lie to Please You: Preventing Biased Feedback Loops in Aasiste- Intelligence Analysis. Analytics, 2(2):350-358, 2023, ISSN 2813-2203. doi: 10.3390/mmalytics2 020020. URL https:t////// / mdpi cwm/2813 221///322222/2222222222222
H. J. A. A. K. M. L. A. B. S. L. W. Tong Mu, Alee Helyar. Rule Based Rewards for Language Model Safety, 2024. URL https: / //t./ openai. bww.----aaaddddddddeee-eeeeeeeee
language-modes model-ooellllaaa safety. pdf. Accessed: 2024-10-28.
H. Touvron, L. Martin, K. Stone, Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models. arxiv e-prints, 2023.
Korenev, P. S. Koura, M.-A.
F. R. Ward, F. Belardinelli, F. Toni, and T. Everitt. Honesty Is the Best Policy: Defining and Mitigating Al Deception. arxiv e-prints, 2023.
J. Wen, R. Zhong, A. Khan, E. Perez, J. Steinhardt, M. Huang, S. Bowman, H. He, and Feng. Language Models Learn to Mislead Humans via RLHF, 2024. URL https://arxiv org/ abs/ 2409. 12822.
S. Wu. Introducing Devin, the first Al software engineer. https://www.t http://wwwww.oniii.oooo.oo ntroducing-den devin, 2024. Accessed: 2024-05-06.
com/i
Zhuang and D. Hadfield-Menelll Consequences of Misaligned AI. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS' 20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
B. D. Ziebart, A. L. Maas, A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In D. Fox and C. P. Gomes, editors, AAAI, pages 1433-1438. AAAI Press, 2008. ISBN 978-1-57735-368-3, URL http:///bb uni trier de/ conf /aaaa/aaaaaaa0e/////// html## iebartMBDO8.
14

---

APPENDIX
In the appendix, we provide more extensive theory, proofs, and examples. The appendix makes free of concepts and notation defined the main paper. In particular, throughout we assume general MDP together with observation kernel Po : + Ω and a human with general belief kernel (ō 3), unless otherwise stated. See the list of Symbols in Section A refresh notation.
In Section we supplement the examples from the main paper with more mathematical details.
In Section D, we provide an extensive theory for appropriately modeled partial observability in RLHF. This can mainly be considered a supplement Section and contains our main theorems, supplementary results, analysis of special cases, and examples.
In Section E, analyze the naive application of RLHF under partial observability, which means that the learning system is not aware of the human's partial observability. This section is essentially supplement to Section 4 and contains an analysis of the policy evaluation function Joha. of deceptive inflation and overjustification, and further extensive mathematical examples showing the failures of naive RLHF under partial observability.
Contents of the Appendix
A List of Symbols
16
B More related work
18
C Details for deception and overjustification in examples
19
C.1 Example hiding failures
19
C.2 Example B: paying to reveal information
20
C,3 Derivations and Further Details for Fig. 4A
20
C.4 Ambiguity in Section 4.4 examples when modeling partial observability
24
C.5 Experimental details
25
Modeling the Human in Partially Observable RLHF
26
D.I
The Belief over the State Sequence for Rational Humans
27
D.2
Ambiguity and Identifiability of Reward and Return Functions under Observation Sequence Comparisons
28
D.3
The Ambiguity in Reward Learning in Practice
31
D.4
Identifiability of Return Functions When Human Observations Are Not Known
32
D.5
Simple Special Cases: Full Observability, Deterministic P33, and Noisy Po
34
D.6
Robustness Return Function Identifiability under Belief Misspecification
36
D.6.1 Some Norm Theory for Linear Operators
36
D.6.2
Application to Bounds in the Error of the Return Function
38
D.7
Preliminary Characterizations the Ambiguity
D.8
Examples Supplementing Section 5
41
Issues of Naively Applying RLHF under Partial Observability
45
E.1
Optimal Policies under RLHF with Deterministic Partial Observations Maximize Jobs
45
E.2 Interlude: When the Human Knows the Policy and is Bayesian Reasoner, then Jobs J
46
15

---

E.3 Proof of Theorem 4.5
47
E,4 Further Examples Supplementing Section 4.4
48
NeurIPS Paper Checklist
54
A List of Symbols
General MDPs
S A(S)
Set of environment states S Set of actions a A of the policy
Set of probability distributions over S. Can be defined for any finite set Transition kernel
8 x A - A(S)
Po E (S)
Initial state distribution Usually the reward function
RED R$
R' E ERS RS
Usually reward function in the kernel of BoT Usually another reward function, e.g. inferred by learning system
Re R
WE 1
Discount factor
T:S + A(A)
A policy
T A(S)
Transition kernel for fixed policy given by T" (s)
EasA s, a) aaa
TEN
Finite time horizon State sequence distribution induced by the policy State sequences E S supported by pa
ps A(ST))
S SC
GE R$
Usually the true return function given by G(3) =
Σ 7 R(ss).
E' RS
Usually a return function in ker B Usually another return function, e.g. inferred by a learning
GE R
system
The true policy evaluation function given by (()) =
Eg-Pa [G(
Additions to General MDPs with Partial Observability
Ω Po:8 A(2)
Set of possible observations E Ω Observation kernel determining the human's observations The observation sequence kernel given by Pa 8)
: + A(?))
IT=o (o st))
The set of observed sequences Ō E Ω²T that can be sampled from ( 8) for 5' Š
0:8-Ω
Observation function for the that Po is deterministic; given by ((s) ) = o with o such that Po(o x)= =
Ō:5-ā
Observation sequence function for the case that P. is deter-
ministic; given by Ö(5) = 5 with ō such that Po 8)
G3 E R (8550(3))0)
Restriction of the return function G 5 to { E $
Ö(5) o) for fixed Ø E Q
Gobs E R
Return function that can be inferred when partial observ- ability not properly modeled, given by Gobe(5)
(B (O(G)
Jobs
Observation policy evaluation function, defined in Eq. (4)
16

---

State- and Observation Sequences
8x S
The l'th entry in state sequence
E'E E S
State sequence 3 = : ST
E St
State sequence segment á s0, 8 for S T
0: EΩ
The 1'th entry in an observation sequence Ø
E QT
Observation sequence o = Ou,...,.0 01
0 O
Observation sequence segment 00...... Or for t S ST
The Human's Belief
((x)
The human's 's policy prior
B(5)
The human's 's prior belief that sequence ø will be sampled, given by (5)) = J B( (5)dx'
B($ ()
The human's 's belief of a state sequence given an observation sequence, see Proposition D.I for Bayesian version The human's 's belief of a state sequence given an observation sequence; it is allowed depend on the true policy æ, see Proposition D.1
())
Bg ( 6
Vector of prior probabilities B()) for 5 { $ ((()) o}
Identifiability Theorem
8>0 0
The inverse temperature parameter of the Boltzmann rational human
a: R 1)
The sigmoid function given by ((x) I+exp(-p)
T: T:RS R
Function that maps a reward function R to the return function ((R) with [TR)] (8) Ei= >² R((se)
B: B:R$ + R
Function that maps a return function G to the expected return function ( on observation sequences given by [ Bg~ [G(5)]
F:RS + R
The composition F Bor
(3 3")
Boltzmann rational choice probability in the case of full observability (Eq. (1))
(2 8¹)
Boltzmann rational choice probability in the case of partial
observability (Eq. (2))
0: R + R
Abstract linear operator given by [()) =
Eg-PE(0 [u(ō]]
2 O R f + R Sx8
Formally the Kronecker product of O with it- self, explicitly given by [OO (5())
Eg,8'~P9(11.5 [C(ā,a")
Robustness to Misspecifications
|
Euclidean norm of the vector x E Rs
A||
Matrix norm of the matrix A. given by || |||||
max,, ||=||=1| || A x|
T(A)
Matrix quantity defined Equation (9)
C(A.p) ())
Matrix quantity defined in Equation (10)
(B)
Restriction B to im
17

---

General Sets and (Linear) Functions
JA|
Number of elements in the set A Intersection of sets A and C Union of sets A and C
AnC
AUC
A
Relative complement of C' in A
82
The Dirac delta distribution of point z in a set; given by
8, (A) = 1 æ E A and (AA) = 0, else
ker A
The kernel of linear operator A V
+ W; given by
ker A = { 0 E A(v)= =0}
im A
The image of linear operator A V + W; given by
im A A= [t E W | E V (v) = w
(v)
Preimage of y under a function ʃ X + Y; given by
(1)( { X f(x) v)
B More related work
Here we extend the related work outlined in Section 2.
A review of limitations of RLHF, including a brief discussion of partial observability, can be found in Casper et al. [2023]. RLHF is a special case of reward-rational choice [Jeon et al., 2020], a general framework which also encompasses demonstrations based inverse reinforcement learning [Ziebart et al., 2008, Ng et al., 2000] and learning from the initial environment state [Shah et al., 2019], and can be seen as special case of assistance problems [Fern et al., 2014, Hadfield-Menell et al., 2016, Shah et al., 3021]. In all of these, the reward function is learned from human actions, which in the case of RLHF are simply preference statements. This requires to specify the human policy of action selection-Bol Bolttiann rationality in typical RLHF- -hhichh can lead to wrong reward inferences when this specification wrong [Skalse and Abate, 2022]; unfortunately, the human policy can also not be learned alongside the human's 's values without further assumptions [Mindermann and Armstrong, 2018]. Instead of model of the human policy, in this paper we mostly focus on the human belief model and misspecifications thereof for the case that the human only receives partial observations.
Related work [Zhuang and Hadfield- Hadfeeld-Menel, 2020] analyzes the consequences of aligning an AI with a proxy reward function that omits attributes that are important to the human's 's values, which could happen if the reward function is based on a belief over the world state given limited information. Another instance are recommendation systems [Stray, 2023], where user feedback does not depend on information not shown which crucially part of the environment. Siththaranjan al. [2023] analyze what happens under RLHF if the learning algorithm doesn' 't have all the relevant information (c.g. about the identity of human raters), complementing our study of what happens when human raters are missing information. Chidambaram et al. [2024] and Park et al. [2024a] deal with the situation that different human evaluators may vary in their unobserved preference types. In contrast, we assume a single human evaluator with fixed reward function, which can be motivated by cases where the human choices are guided by behavior policy, constitution, or a model spec [TTng Mu, 2024, Anthropic, 2023b, OpenAl, 2024b]. Kausik et al. [2024] assumes that the choices of the human evaluator depend on an unobserved reward-state with its own transition dynamics, similar to an emotional state in real human. In contrast, we assume the human to be stateless.
Finally, we mention connections truthful AI [Evans al., 2021, Lin et al., 2022, Burns et al., 2023, Huang et al., 2023], which is about ensuring that AI systems tell the truth about aspects of the real world. Partial observability caught: If the human evaluator does not observe the full environment, or does not fully understand it, then they may not detect when the Al is lying. More speculatively, we can imagine that AI models will at some point more directly influence human observations by telling us the outcomes of their actions. E.g., imagine an Al system that manages your assets and assures you that they are increasing in value while they are actually not. In our work, we leave this additional problem out of the analysis by assuming that the observations only depend on the environment state, and not directly on the agent's 's actions.
mechanism that makes feasible for models to lie without being
18

---

C
Details for deception and overjustification in examples
A. Suppress errors with 2> /dew/null
B. Expose successes with -verbose flag
P
L
O(L)
ac
0
W
ap
0
W
+
S
[1
3
S
41
DIW)
[1S]
RH
((1)
Wi
47
OIS)
QV
OL1)
Wv
47
4'7's
0,f
477's
aV
0106a 5p
Grai1
L
07
Ly
O
08
(i - 0g
Figure 6: Two example MDPs with observation functions in which RLHF chooses undesirable policies. Each box depicts a state with a footer showing the (deterministic) observation produced by that state. Outgoing edges from each box are available actions. A more detailed diagram for the first MDP, with explicit shell commands and log messages, available in Appendix C.3.
Here we include details to the examples described in Section 4.4 that illustrate the failure modes of RLHF in the presence of partial observability. For each of the following, we will characterize the policy which maximizes Jobe. as this is the policy RLHF selects for when observations are deterministic; see Proposition 4.1.
Our examples feature an agent trained with RLHF to complete tasks user's terminal. The output of each command (stdout and stderr) is piped to a log file, which is what the human evaluator sees when making choices for RLHF We assume that the final state T has a self-transition, episodes have a fixed horizon length of 3 (meaning state sequences have length 4: 8o.... 88),
1, and there
is a fixed initial state 50 = S. Both examples feature a fixed transition probability < 1. We assume that the human's 's belief only supports possible explanations: B( ō) > 0 = ((3) =
For further details about these examples, including tutorial-style analysis of Example A, see Appendix C.3.
C.1 Example A: hiding faallures
See Appendix C.3 for derivations and tutorial-style analysis of this example.
In Fig. 4A (and Fig. 6A), the agent's 's task is install CUDA, (states W and WH) which requires first installing Nvidia drivers (state I). If the CUDA installation fails, one enters L or Lμ- The ac action is a command to install CUDA that logs both successes and failures, whereas asr corresponds to aç with 2> /dev/null appended, which suppresses error messages such that no log message is produced the installation fails. The human may prefer that the agent not hide failure; we price this into the reward function with penalty r 2 0 on the reward at LH. The agent may attempt the CUDA installation before installing drivers, but this will fail.
There are three pairs of trajectories which produce identical observations. Here we address the most prominent (see Appendix C.3 for the others): SITT and SIL both produce ogorogoy. stylized as a log containing only success confirmation for Python (Fig. 1, ō2). after successfully installing drivers, a failed CUDA installation with 2> /dev/null (SIL,; and simply exiting (SITT) both produce log containing only a success confirmation for the drivers (ogoropog). Let Po == ( STL T Ø = og0))g00) E (0, 1) be the human's 's suspicion, upon seeing only a success confirmation for the drivers, that failed CUDA installation was hidden.
We consider two cases which differ only in whether the expected value of attempting to install CUDA (with default logging, ac) after installing the drivers is positive. In Example C. the success probability p high enough that the true optimal policy to attempt the installation; in Example C.2, p is low enough that the optimal policy is to terminate.
policy n" is to attempt to install CUDA with default logging (ac) after installing drivers, but PH < 547, then the maximiziig policy, which will be chosen by RLHE to instead use the 2> /ew// /ulll trick. Intuitively, this because if the installation fails, the human is likely to believe that the agent just didn't 't attempt install CUDA, which is preferred
Example C.I. If the true-reward-opiin
19

---

failed installation. This is deceptive inflation: the overestimation ervor E" for this policy is positive, whereas ET (T²) = 0, in line with Theorem 4.5.
5_, If PH 5+r then RLHF will select n².
Example C.2. a is to terminate (ar) after installing the drivers, but pH < learn the deceptive inflation 2> /dev/null policy as in Example C.I.
5_ 34p then RLHF will
5_ RLHF will choose the policy that attempts to install CUDA with default logging. Intuitively, this is because the combination of increased suspicion that the agent might be hiding errors (higher pa) and increased penalty on this suspected behavior (higher r) also penalizes our preferred trajectory SITT since it can be misinterpreted as hiding errors. This is overjustification; the overestimation error E is 0 (thus this is not deceptive inflation), and as required by Theorem 4.5, the underestimation error E is O, lower than E (TT") (5
However, if PH > 54T
C.2 Example B: paying to reveal information
In Fig. 4B (and Fig. 6B), the agent's s task is to install Python (state 1) and to optionally further install NumPy (states W and Wv). The ap action corresponds to command to install NumPy with "default" settings which only logs errors, whereas @v corresponds to the same command with a --verbose flag that adds additional info. In the case of success, the human distinctly prefers not to see this verbose output; we price this into the reward function with a penalty r > O on the reward at Wv.
There only one pair of trajectories which produce identical observations: after successfully installing Python, successful NumPy installation with default logging (S1WT) and simply exiting (SITT) both produce a log containing only a success confirmation for Python (og0rogoo). Let PD = B33 = S/WT ō o30r0gos) E (0, 1) be the human's 's optimism, upon seeing only success confirmation for Python, that NumPy was also successfully installed (without the -- --verbose flag).
Here we consider only the case where is large enough that the true optimal policy is to install Python then attempt install NumPy with default logging (ap).
Example C.3. If x² is to attempt to install NumPy with ap after installing Python, and p( r) then RLHF will select the policy that terminates after installing Python. Intuitively, this is because the agent can exploit the human's 's optimism that NumPy was installed quietly without taking the risk of an observable failure (L). This is deceptive inflation, with overestimation error E 5pp. greater than E. (r*)=
PD > >=:
If instead PD < q. then RLHF will select the policy that attempts the NumPy installation with verbose logging (av). Intuitively, this is because the agent willing to "pay" the cost of r true reward to prove to the human that it installed NumPy, even when the human does not want to see this proof. This is overjustification; the overestimation error E. is (thus this is not deceptive inflation), and the underestimation error E is O, lower than E (TT" 5p(1 -PD).
C.3 Derivations and Further Details for Fig. 4A
> apt install cuda
Upigi: processing
AC apt Install cuda
W
10
S
apt instalt mvidia-driver
(No outsut?
0
Inatalled scidia-driser
Wi
(''1
exit
20 installed cuda
Apt install cuda /drv/aul
1.11
( 9
apt install cuda 2> /dww/mull
9g [No outqut)
Figure 7: An expanded view of Figure 4A. Commands corresponding to the various actions are depicted along edges, and log messages corresponding to the various observations are depicted underneath each state.
20

---

We first include Figure 7, a more detailed picture of the MDP and observation function in Ap- pendix C.I, help ground the narrative details of the example.
Next we formally enumerate the details the MDP and observation function.
S [5,1, W, WH. LH,
A {ar: "C, "H ar).
' is depicted in Figure 7 and Figure 4A. For state 8, any outgoing arrow labeled with an action a (such as ay) describes the distribution T(s' 8, a) as follows: if the arrow does not split, then T(s') 8, a, then for each successor state s' it eventually reaches, a probability is written just before the box corresponding to s' (for this example,
where s' is the state the arrow points to; if the arrow does split,
p), and T s, - q. Additionally, any action taken from state that does not have outgoing arrow corresponding to that action will immediately transition to state T, as though ar had been taken.
p or q
Any action taken from state T transitions deterministically to (o( 1.
as described in the table (the numbers in the top right of each state box) with r 2 0. Additionally, R(S) = R(T) 0.
1.
We work with a fixed horizon length of meaning state sequences have length (since time is zero-indexed: 8p818283).
The observation function is also depicted in Figure 7. Each state deterministically produces the observation in the lower-right corner of its box in the figure. We also write it in another format in Table 9.
Table 9: The observation function for the example in Appendix C.I and Appendix C.3.
S
I
W
W
H
L
LH
T
(())
OW
OW
OT
0g
0g
We make the additional assumption that the human belief B( ō) only supports state sequences 5
which actually produce ō under the sequence observation function Ō: ( In particular, this means that for any ø which is only produced by one 5, ( 8) 1.
> 0 = Õ(5) =
There are three pairs of state sequences which produce identical observation sequences. For each, we introduce a parameter representing the probability the human infers the first of the pair of state sequences upon seeing their shared observation sequence.
1. STLHT and STTT both produce 0g0r0808. a log containing only success confirmation for installing drivers, again because O(TH) = O(T) = Og. Let PH B STT.
Ø og0ro)o0).
2. STTT and TT both produce op00000g empty log, since O(Ih)) = 0(7) = 0g-
Let 1/1 B(8 38==Spy ST ō== ogogogoo).
3. STWT and 7 both produce ogorow Og. a log containing success confirmations for installing both drivers and CUDA. Let pw = B(g STWT ō Og0row
We assume for simplicity that p'y PH. i.c that the human is just as likely to think an empty log following successful driver installation contains error that was hidden with 2> /dev/null (PH), as they are to think that an entirely empty log contains a hidden error (p'r). In principle, this need not be true: the human may have differing priors about the agent's behavior in the two different circumstances. However, the algebra to reason about such case significantly more cumbersome, and this case reveals fundamentally different agent behavior under framework that isn't present in some simpler case.
We can thus write the full B as matrix as in Table 10.
21

---

Table 10: The parameterized human belief function B for the example in Appendix C.I and Ap- pendix C.3, expressed as a matrix (rendered as a table). Any empty cell is equal to 0.
STTT
SL
SLTT
SITT
SIL
SILT
SIWT
SIWHT
Og
pı
PH
0g0r,000p
Og0r0g00
PH
PH
00010100
Og
PW
- PW
We have laid the groundwork sufficiently to begin reasoning about the observation return, overesti- mation and underestimation eπrоr, policies which are optimal under the reward function learned by naive RLHE, and the resulting deceptive inflationand overjustification failure modes. We begin by computing the measures of interest for each state sequence, shown in Table 11.
Table 11: Measures of interest for each state sequence for the example in Appendix C.1 and Appendix C.3. State sequences which produce the same observations have their Gohs columns merged, since they necessarily have the same Gohs-
8
G(
GG(5)) PHG(SLn (1 pm G(STTT) = =-pr +r)
Gobe ()) = gg (1)
E+ = max(O,
E (3) = max 00 G( Gota (3)
Gobs 3) G(3)
STTT
pH + r)
SLuTT -5
pH +
SLTT -5
SITT
pH G(SITT) PHG[SIL#I] (5 +r)
0 (1 PH) (5 +
PH (5 + r)
SILST
-4 --
SILT -4
-4
0
SIWT
11
GWSIWT (1 (1- Pw G((SIHH))
0
SIWHT T
II
As an exercise, checking the computations in Table 11 is quick way to gain some intuition for how these quantities relate. It's further useful to speak about these quantities using their names, and work through the stories these expressions tell.
Consider the E (TL cell. What does it mean that this is (1 (1-PH)(C P5 r)? E+ is the overestimation error; (1 PH (5 +r) is the expected amount by which the human observer, upon seeing a completely empty log (o94)))09)) will overestimate the total reward the agent attained when those empty logs were actually produced by SLuTT. This is trajectory in which the agent immediately uses the 2> /dev/null trick, fails to install CUDA without logging the error, and terminates. Under what circumstances might the human overestimate the total reward when 5== SL#TT? Upon seeing the empty log, the human thinks with probability 1 1 PH that the agent simply terminated immediately, which would be worth total reward of O. Since the actual total reward 5 r, this an overestimate by 5 + r. The human thinks with probability pH that SLn occurred. This is correct, there is overestimation and this pH does not contribute to the overestimation error. So we have that with probability 1 PH. the human overestimates the total reward by +r.
We can keep going! Why is the underestimation error S/WT equal to 0? Because the only other trajectory with which it can be confused attains the same total reward, so regardless of how the probability mass of the human's 's belief divides between them, there will be no underestimation. Can all of the zeros in the overestimation and underestimation error columns be explained this way?
We now move on consider policies rather than state sequences. Since policy π imposes distribution pr over state sequences (the "on-policy distribution"), our policy measures are in fact exactly parallel to our state sequence measures. Each one is an expectation over the on-policy distribution of the columns of Table 11. We restrict our attention to deterministic policies which only take actions depicted in Figure (i.e. that never terminate via an action other than aT), which there are only six in this MDP. They are enumerated, along with the policy-level measures, in Table 12. Policies will be written as sequence of actions enclosed brackets, omitting trailing repeated ar actions. This is nonstandard notation in an MDP with stochastic transitions, but is unambiguous in
22

---

this example, because all decisions are made before any stochasticity occurs. The policies are [er], [rn) [acar], [ayar], [αyanar], and [aTaço]].
Table 12:
Measures of interest for each policy for the example in Appendix C.1 and Appendix C.3. Each of the columns here is the on-policy average of the corresponding column in Table 11. Policies are written as sequences of actions, omitting trailing repeated aT actions. This is nonstandard notation in an MDP with stochastic transitions, but is unambiguous in this example since all decisions are made before any stochasticity occurs.
J (TT
Jobs (TT)
E (T)
E (T)
4T
PH (5 +
PH +
PH (5 + r)
(1 PH (5 + r)
aC0T
5
5
0
0
010r
PH +r)
0
PH (5 + r)
PG(SIWHT)
pGoba HT)
[ayaнar]
+(1 - ((I-))G(SLLR))
+(1 (SILLH)
(1 - p) (1
0
11 (1-(11))(15+) +r)
11 p) [10 (5
PG(S/WT)
PG (SIWT)
[araçar]
+(1 - ((SLLTT
+(1 - Gbbs ( (SILT) 11 15
11 11-
With this have everything we need to characterize optimal policies under the reward function learned by a naive application RLHF ("policies selected by RLHF"). By Proposition 4.1, we know that if Po is deterministic, in this example, RLHF selects policies which maximize Jobs- In order to understand the behavior of these policies, we'll also need to determine the true optimal policies, iee those which maximize J. We'll proceed in cases, only considering boundary cases (specific measure-zero parameter values for which the result is different) insofar as they are interesting.
Case 1: 3.4 If > 3. the CUDA install (with default logging, ac) is likely enough to succeed that it's worth attempting it: p- R(W) + (1 - p) R(L) > 0. also immediately follows that [(araçar)) Jobss (aaass((())
((rracar) J
p) 15 1.
This allows us to eliminate policies [ar], [agar], [acar], and [a,ar], which all have J S 1 and Jobs S None of them can thus be J-optimal or Mobs-optimal All that remains is to compare J and Jobs for [araæar] and [araçat]. We can check the sign of the differences of these pairs of values, starting with J.
J [[araaar) [(araaar))
-p)r
Since p is a probability and r is nonnegative, this value is positive (and thus [aιacar] is preferred to [aramar]] by the human) if and only if p < and 0.
Jaba((aaaaar) - Jobs (absaaa))aar))
p) (5 (((5(+)) +r]]-
This value is positive (and thus [arанar] is the policy RLHF selects) if and only if p < and 5 Pi 5+r
p = 1, then both differences are O, and both J and Johe are indifferent between the two policies. This makes sense, they differ only in the case where the CUDA installation fails; this happens with probability 1 p O when p 1. Now suppose p < 1. If r O, then the human is indifferent between the two policies. This also makes sense, as r meant quantify the extent to which the human dislikes suppressed failures; if it's zero, then the human doesn't It care. However, ps < 5+r, then Johs (arraaa)))))) > Jats((urunar)), and thus RLHF favors the 2> /dev/null policy [а]анar].
If p 1, > 0. and Pa < 54r- then we have that [[aracar]) > ][araner) but Johs (Ba ((raa)a))) > Jars([aranaa]] Thus RLHF will select the 2> /dev/null policy [аraneт], and by Theorem 4.5, since [ayanār] is not J-optimal, then relative to (araçar), it must exhibit deceptive inflation, overjustification, or both. Intuitively, should be suspicious that deceptive
23

---

inflation is at play whenever the agent hides information from the human. Indeed, referencing
Table 12, we have E= ([aragar)) = (1 (1-p)(1- Per (5 + > 0 = E ([aracar]) Together with Jobs ((aaagar))) > Jo (lajacar)) }), this satisfies the conditions of Definition 4.3, and thus this is an instance of deceptive inflation.
p < 1,r > O, and pi > 84p. then [asacar] is optimal under both J and Johs. and in this case, RLHF selects the true optimal policy.
5
Case 2: 1. In this case, the CUDA install is not likely enough to succeed to be worth attempting (under the true reward function). Mathematically, J(Jaaaar)] S I(aaracrr) < 1 = ][{ [[]aaar]). The other three policies are always worse under J than [ajar], so we have our optimal policy π² = [arar]. However,
3:
Jobs (araaa))))
((aaar)))
( (5))))
which is always positive unless p = 0. As long as p > 0, i.e. installing CUDA is actually possible at all, then regardless of everything else, Johs (aa((aard ([aragaT]) > Joha( (oha((arar)). We can interpret this reasonably: however good or bad it might look terminate after installing drivers (SITT), using the 2> /dev/null trick will either produce the same observation as SITT (and thus the same feedback distribution) or will actually visibly install CUDA successfully- the most lucrative possible source of positive feedback. Even if there's 's only a sliver of a chance of success and it's a catastrophe to have a silent failure (i.e. r >> 0). however likely you are to infer that the lack of a log message about CUDA means the agent hid failure, it will penalize [arar] more than it will penalize [ayaнōr]: where [aragar] produces this ambiguous observation with probability p (when it fails), [arar] will always produce the ambiguous observation.
This means that when p < 3. it is impossible to recover the true optimal policy with naive RLHE Which policies can possibly be Jats-optimal for some setting of the parameters? We can similarly rule out [ar] and [anar] for 0 < p < 3:
(oba((aragar)) Jobe (laaar)) = p(10 + PH (5 + r)) > 00
We can rule out [acat] by comparison to [araçar): Johs ((araçar)) Jahs ((aa(()))))) = 16 (1 p)15 > O. So we are left with only [araнeт] and [aιaçar] as candidate Johs optimall policies.
As in Case 1, we find that Jubs (araa(a))))))) > Jots((arar)) if and only if p 1 or pH < 54r* In
case 2 we have assumed p < 1, leaving only the pH condition.
3.
If pH π² = [a because
then RLHF selects [araнar]. As in Case 1, this is deceptive inflationrelative to
E ((aagarr) = (1- P5)
(x').
If PH 5+r then RLHF selects [aιacar]. Because this policy is not J-optimal, by Theorem 4.5, we must have deceptive inflation, overjustification, or both. Which is it? Here the optimal policy is to terminate after installing drivers, [arar]- However, PH
5
5 5+r This can be rewritten as ( r) > 5. We have seen this expression (5 + r) before; it is the underestimation error incurred on 5 = SITT and therefore also the average underestimation error of policy [oyar]. So here the underestimation error on the optimal policy- that is, the risk that the human misunderstands optimal behavior (terminating after installing driver) as undesired behavior (attempting a CUDA install that was unlikely to work and hiding the mistake)-is severe enough that the agent opts instead for (a) worse policy that attempts the ill-fated CUDA installation only to prove that it wasn't t doing so secretly. qualitative terms, this quintessential overjustification behavior. Indeed, relative to reference policy x' [aTaT], we have
E ((racar)) = =0 0 < pn +r) = F (m²)
J [(aracar]) 11 -1100 (1 p) 15 < 1 = J(x'),
and thus by Definition 4.4, this is overjustification
C.4 Ambiguity in Section 4.4 examples when modeling partial observability
Consider the example Fig. 4A when modeling partial observability as in Section By Theorem 5.2, the ambiguity in the return function leaving the choice probabilities invariant is given by ker B 0 im I.
24

---

po-aware an Table 13: Experiments showing improved performance of po-aware RLHF
Ex.
p
Phide
Pdefault
model
action
E
dec. infl.
E
overj.
optimal
A
0.5
0.5
N/A
naive
4n
1.5
0
X
X
0.5
0.5
N/A po-aware
a
1.5
/
x
X
A
0.1
0.9
N/A
naive
aC
x
0
V
X
A
0.1
0.9
N/A
po-aware
4r
0
5.4
x
V
0.5
N/A
0.9
naive
4T
4.5
V
x
0.5
N/A
0.9
0.25
V
0.5
N/A
0.1
naive
4v
V
B
0.5
N/A
0.1
po-aware
a
0
X
2.25
X
Let R O0, (WW), ((((n).0.O) E R R(S....WW. be reward function that want to parameterize such that G" = P-R ends in the ambiguity; here, R' is interpreted as a column vector.
We want B G = O. Since the observation sequences ō= ogogogop. ō == 080r ō o= ogorogos. o og all cannot involve the states W WH, f, is clear that they have zero expected return (B) Set F' == B B(SIWH') H (g0rowop). Then the condition that B -G" = 0 is equivalent to:
0= 0 (B" (0g0rooo)))) (3
E
[+](T]
= = V''SG"(STWH1T + (1- (00-P')-"(SSWTW))) ('ISTT) = F'H R'(Wn) + (1-P'H) (W).
Thus, if (WW) - (WHH) then G" ker B Nim I, meaning that R R+-R' R²' has the same choice probabilitics as R and is thus fully feedback-come combbabb... In particular, if (WWW)) > 0 sufficiently large, then in subsequent policy optimization, there is incentive to hide the mistakes and ли will be selected, which is suboptimal with respect to the true reward function R.
Thus Fig, 4A still retains dangerous ambiguity when modeling partial observability.
However, the example in Fig. 4B leads ambiguity when partial observability is correctly modeled.
To show this in detail, let G' T(R') ker B B I. We need to show G" = 0. Since the human only uncertain about the state sequences corresponding to the observation sequence Ogo the condition B G already implies G' (*) 0 for all state sequences except SIWT and STTT. From (B G²) (o80000)))))) = 0, one then obtains the equation
(1 (1-PD)- (R'(S) RT(1)) + 2R' (T) + +p (I'(() + (2(
= 0. (5)
Thus, if one of the two state sequences involved has zero return, then the other has as well, assuming
that 0 * PD * and we are done.
To show this, R²(S) (L) + 2/ 2R'(T). (T), from which (L) (T' follows. Then, from R'(S) + ((( substituting the previous result gives R((S)) R'([) + (T) 0. and so
that all other state sequences have return: R'(S) + 3R)(T)
R'(L) RT)
Equation (5) results in R((S)) + RR(T)) + (')))+ + (T) = 0. Overall, this shows G' [T(="'(> and ker B B im I (0).
C.5 Experimental details
Here, we explain more experimental details for the results in Table 1, reproduced here as Table 13. The leftmost column ("Ex." for "example") corresponds to Examples A and B in Figure 4. is the success probability upon attempting to install Cuda or NumPy in state 1, see Figure 6. Phide in Example A is the human's 's belief probability that the agent hid the error message if there is no output after nvidia-driver installation. Similarly, Paefauh Example B the human's 's belief probability that installation was done with default settings if there is no further output after Python
25

---

installation. Note that lines one and two in the table also correspond to Example C.I, lines three and four to Example C.2, and lines five and six to the first half and seven and eight to the second half of Example C.3, respectively. In all our experiments, we set the penalty to r
The "model" column has value "naive" if the reward learning algorithm is classical RLHF (erroneously assuming full observability) as in Christiano et al. [2017]. and "po-aware" if the human's 's partial observability is correctly modeled in Appendix D.3. We initialize the reward function as list of rewards of states and train it by logistic regression using a dataset that consists of all pairs of state sequences together with the human's choice probabilities under partial observations. This leads to 28 pairs of distinct trajectories together with choice probabilities. We train the reward model for 300 epochs over shuffled dataset of 13.5 copies of the 28 pairs with the Adam optimizer, for a total of 1 113400 training updates.
Once we have the resulting reward model, we use value iteration to find its deterministic optimal policy. All policies choose to install the nvidia-driver (in Example A) and Python (in Example B), and differ in their action state I, which is given in the column "action". We compute the overestimation error and underestimation error of the resulting policies analytically using the hardcoded environment dynamics, true reward function, observation function, and human belief matrix B. This is given in columns E and E' Note that these are averages over 10 entire training runs, though since they always result in the same learned policy, there no variation and we do not state any uncertainty.
The columns "dec. infl.", "overj.", and "optimal" state whether deceptive inflation or overjustífication occurs with the learned policy, and whether it is optimal according to the true human's 's reward function.
D Modeling the Human in Partially Observable RLHF
In this appendix, we develop the theory of RLHF with appropriately modeled partial observability, including full proofs of all theorems.
In Section D.I, we explain how the human can arrive at the belief ( ō) via Bayesian updates. The main theory and the main paper in general do not depend on this specific form of the human's 's belief, but some examples in the appendix do.
In Section D.2 we then explain our main result: the ambiguity and identifiability of both reward and return functions under observed sequence comparisons. In Section D.3, we then explain that this theorem means that one could in principle design a practical reward learning algorithm that converges on the correct reward function up to the ambiguity characterized in the section before, if the human's 's belief kernel (5 ō) is fully known.
In Section D.4, we generalize the theory to the case that the human's 's observations are not necessarily known to the learning system and again characterize precisely when the return function is identifiable from sequence comparisons. We then consider special cases in Section D.5, where we show that the fully observable case is covered by our theory, that a deterministic observation kernel Pa usually leads to non-injective belief matrix B, and that "noise" in the observation kernel Pa leads, under appropriate assumptions, to the identifiability of the return function.
Our identifiability results require that the learning system knows the human's 's belief kernel B(5 Ø). In Section D.6, we then show that these results are robust to slight misspecifications: a bound in the error in the specified belief leads to a corresponding bound in the error of the policy evaluation function used for subsequent reinforcement learning.
In Section D.7, we then provide a very preliminary characterization of the ambiguity in the return function under special cases.
Finally, in Section D.8, we study examples of identifiability and non-identifiabiiity of the return function for the case that we do model the human's partial observability correctly. This reveals qualitatively interesting cases of identifiability, even when B not injective, and catastrophic cases of non-identifiabiiity.
26

---

D.1 The Belief over the State Sequence for Rational Humans
Before we dive into the main theory, we want to explain how the human can iteratively compute the posterior of the state sequence given an observation sequence with successively new observations. This is done by defining Bayesian network for the joint probability of policy, states, actions, and observations, and doing Bayesian inference over this Bayesian network.
The details of this subsection are only relevant for few sections in the appendix since it is usually enough to assume that the posterior belief exists. Additionally, in the core theory, we do not even assume that (3 o) is a posterior: it simply any probability distribution. The reason why it can still be interesting to analyze the case when the human is a rational Bayesian reasoner is that one can then analyze RLHF under generous assumptions to the human.
We model the human to have a joint distribution ( 8, a. o) over the policy x, state sequence S0, .... Sy, action sequence du .... 41-11 and observation sequence o 00....:00 Or. This is given by a Bayesian network with the following components:
a policy prior B(T');
the probability of the initial state (ss)) Po(so):
action probabilities B(a x, a(a 8);
transition probabilities B( 8;; (1) = ( 81, a:);
and observation probabilities (( (t) Po 8t).
Together, this defines the joint distribution 5,,,,, ō) over the policy, states, actions, and observa- tions that factorizes according to the following directed acyclic graph:
n
SU
0g
Q1
$2
a2
$3
(6)
OU
01
02
03
The following proposition clarifies the iterative Bayesian update of the human's posterior over state sequences, given observation sequences:
Proposition D.1. Let t < T - and denote by š 80. ... St state sequence segment of length t # O. Similarly, .u....: denotes an observation sequence segment. We have
B(S S1-1.7 6, 0+11 & Po (841))
Σ T(st1 8,, at) T(aa |s))
BB3, ()).
Thus, the human can iteratively compute B(3, n 0) from the prior B(so, T) = Po(so) B(T') using the above Bayesian update.
The posterior over the state sequence can subsequently be computed by
0) = B(8, 1
Proof. The proof essentially just Bayes rule applied the Bayesian network in Equation (6). We repeatedly make use of conditional independences that follow from d-separations in the graph [Geiger al., 1990]. More concretely, we have
B(s, s1+1, à 0.441) x B (0co11 8, 81411 T, 6) B(3, s1+1. 10)
27

---

Return Functions (on state sequences)
B
[[[eeturn functions obs sequences)
im n ker B (the anbiguity)
Gl
im
B B(im T)
G(
G (im ker B)
(Return functions
ker B
anbiguous with 64)
B (C (im (im Ker
G(FA)
G
B B(ker B)
ker B
G
im r
Figure 8: The linear geometry of ambiguity for hypothetical example with three state sequences and two observation sequences. G* is the true return function, and "G"" is used in labeling the axes to refer to some arbitrary return function. This is a more accurate geometric depiction of the middle and right spaces in Figure 5. The subspace im I n ker B (purple) is the ambiguity in return functions, meaning that adding an element would not change the human's expected return function on observations. Thus the set of return functions that the reward learning system can infer is the affine set G (im P ker B) (yellow). Note that the planes on the left are drawn be axis-aligned for case of visualization; this will not be the case for real MDPs.
Po S(41)
B BBs++1 (8, B(s,
Po 8(+1):
Σ B(s ae 0,, B(ac 8, ,)
BB(s π
aGEA
= Pol 1841)
Σ a.EA
T(1+11 (6, क (a 81)
क 10).
In step we used Bayes rule. step 2, we made use of the independence oc+ oc+ 11 (8, n. ô) st+1+ plugged in the observation kernel, and used the chain rule of probability to compose the second term into a product. In step 3, we marginalized and used, once again, the chain rule of probability. step 4, we used the independences 8(+1 II (sp, ... 81- π, 0) (st, a) and II (8o. 8, (TT, 8t) and plugged in the transition kernel and the policy.
The last formula is just marginalization over the policy.
D.2 Ambiguity and Identifiability of Reward and Return Functions under Observation Sequence Comparisons
In this section, we prove the main theorem of this paper: characterization of the ambiguity that left in the reward and return function once the human's Boltzmann-rational choice probabilities are known. We change the formulation slightly by formulating the linear operators "intrinsicalll" in the spaces they are defined in, instead using matrix versions. This does not change the general picture, but is a more natural setting when thinking, e.g., about generalizing the results to infinite state sequences. Thus, we define S
R as the linear operator given by
[BG (ō) z B(a
E [G()).
Here, is the human's belief, which can either be computed as the previous subsection or simply be any conditional probability distribution. Similarly, we define I -RS + R as the linear operator given by
T
(3)) = Σ 7' R(ss)
1=
28

---

The matrix product B I' then becomes the composition B RS - R². Finally, recall that the kernel ker A of a linear operator A is defined as its nullspace, and the image im A as the set of elements hit by A. We obtain the following theorem:
Theorem D.2. Let R be the true reward function and R another reward function. Let G r (R)) and G = T(R) be the corresponding return functions. The following three statements are equivalent:
(i) The reward function gives rise to the same vector of choice probabilities as R, i.e
(PR(O> σ'))
(PR 5')) 8,0e
8.0'
(ii) There is reward function R' ker(Bo r) and a constant E R such that
R = R R' +c.
(iii) There is return function G' ker B 2mi I and constant d' E R such that
G
In other words, the ambiguity that is left in the reward function when its observation- based choice probabilities known is, up additive constant, given by ker(B (B I); the ambiguity left in the return function is given by ker B n Nim r.
Proof. Assume (i). To prove (ii), let a by the sigmoid function given by (xx T+expl xx) Then by Equation (2), the equality of choice probabilities means the following for all 5' E c
(B ( ([B (o))) - a ( |B(())
Since the sigmoid function is injective, this implies
[B( [B( (7)) (7)) = [o) - [
Fixing an arbitrary o", this implies that there exists constant c' succh that for all ō 2 the following holds:
BBC) BBG)) -'' 0.
Noting that B(c)) = c', this implies & - G - c' E ker ker(B). Now, define the constant reward function
c= c' 1-
1- 1
We obtain
[T(c) = Σ 7d
1=
1 1-7
d'
1
x
T+1
t=
=d.
Thus, we have
r - R c) = G G e' ker(B),
implying R' = R
ker(B o r). This shows (ii).
That (ii) implies (iii) follows by applying I both sides of the equation.
Now assume (iii), i.e. G G+G'++ +e' for constant e' E R and return function G' E (er(B))im
This implies BB²)- BGG) c. Thus, for all ō, σ2 we have
[B( - [B] (7") ( [B]
[B())]
which implies the equal choice probabilities after multiplying with | and applying the sigmoid function both sides. Thus, (iii) implies (i).
29

---

Corollary D.3. The following two statements are equivalent:
(i) ker(BoT)= =0.
(pR(ōz 0'))
(ii) The data
determine the reward function R. up additive constant.
G
Proof. That (i) implies (ii) follows immediately from the implication from (i) to (ii) within the preceding theorem.
Now assume (i). Let Rt E (Bo Define R == R + R'. Then the implication from (ii) to (i) within the preceding theorem implies that R and R have the same choice probabilities. Thus, the assumption (ii) in this corollary implies that a constant. Since I and B map nonzero constants to nonzero constants, the fact that R' ker(B implies that R 0, showing that ker(B e r (0}.
As mentioned in the main paper, the previous result already leads to the non-identifiability of R whenever T is not injective, corresponding to the presence of zero-initial potential shaping (Skalse et al. [2023], Lemma B.3). Thus, we now strengthen the previous result so that it deals with the identifiability of the return function, which is sufficient for the purpose of policy optimization:
Corollary D.4. Consider the following four statements (which can each be true or false):
(i) ker B {0}.
(ii) ker BoTT) = {0).
(ii)) ker Bm (0)-
(iv) The data (pR(ō> o¹)) a. up to a constant independent of s.
determine the return function G ((R) on sequences 5
Then the following implications, and no other implications, are true:
(1)
(iit)
(iv)
(ti)
In particular, all of (i), (ii), and (iii) are sufficient conditions for identifying the return function from the choice probabilities.
Proof. That (i) implies (ii) is trivial. That (ii) implies (iii) is simple linear algebra fact: Assume (ii)
and that G' E ker B im P. Then G' = T(R²) for some I²' E R and
0 B(G')
(T(R)) (B (R)))
By (ii), this implies R' = 0 and therefore G' = T(R)) 0, showing (iii).
That (iii) implies (iv) immediately follows from the implication from (i) to (iii) in Theorem D.2.
Now, assume (iv). To prove (iii), assume G' E ker Bmim r. Then implication from (iii) to (i) in Theorem D.2 implies that G + G' induces the same dbservation-basee choice probabilities as G.
Thus, (iv) implies G + G'
for some constant c', which implies G' é'. Since G' ker B,
this implies 0 = B(G')= (cc') c' and thus G' 0. Thus, showed ker B 3m {0).
30

---

We now show that no other implication holds in general. Example D.32 will show that (ii) does not imply (i). We now show that (i) does also not imply (ii), from which will logically follow that (iii) does neither imply (i) nor (ii). Namely, consider the following simple MDP with time horizon T 1:
b
In this MDP, every state sequence starts in a, deterministically transitions to b, and then ends. This means that ø = ab is the only sequence. Now, let R' E (R be the reward function given by
-1
R'(a)= =1, R'(b) 7
We obtain
-1
[T(R)] (())
- 0. 7
Thus, [R') 0, (B+)) 0, and, therefore, ker r) * [0] Thus, (ii) does not hold.
However, it is possible to choose (5 ō) such that (i) holds: e.g., if f S and B( ō) 65(5).
then ker B = {0} since this operator is the identity.
D.3 The Ambiguity in Reward Learning in Practice
In this section, we point out that Theorem D.2 is not just theoretical discussion: When B and the inverse temperature parameter 3 are known, then it is possible to design reward learning algorithm that learns the true reward function up to the ambiguity ker(B T) the infinite data limit. In doing so, we essentially use the loss function proposed in Christiano et al. [2017].
Namely, assume D is a data distribution of observation sequences 5 E á such that all sequences in Ω have a strictly positive probability of being sampled; for example, could use an exploration policy and the observation sequence kernel Pg. For each pair of observation sequences (ō, (", we then get a conditional distribution P(u| ã, o²) over a one-hot encoded human choice μ E {(1, 0), (0, 1)}, with probability
P(H (1, 0,0') =PR( o').
Together, this gives rise to dataset (ō,, "1,,) ...., (ōx, o'x pr) of observation sequences plus a human choice.
Now assume we learn reward function Rg : R. that differentiable in the parameter 0 and that can represent all possible reward functions R E R$. Let Gg == I(Ra) be the corresponding return (1) (2)). As in Christiano et al. [2017], we define its loss over the dataset function. Write μı = (p4 Mk above by
[(0)
Σ k=1
(1)
(2) log R (ōx > o%) + M
log nao (5% ok).
M
Note that by Equation (2), this loss function essentially uses B and also the inverse temperature parameter 3 in its definition. This means that these need to be explicitly represented to be able to use the loss function in practice.
Proposition D.5. The loss function İ is differentiable. Furthermore, in the infinite datalimit its minima are precisely given by parameters 0 such that Ro R + R' + e.for R' ker (BoT) and E R, equivalently Go G + G' q' + é for G' ker B Boim I and d E R.
Proof. The differentiability of the loss function follows from the differentiability of multiplication with the matrix B, see Equation (2), and of the reward function Rg in its parameter 0 that we assumed.
For the second statement, let ( σ²) be the number of times that the pair (ō, o²) appears in the dataset, and let N( o², 1) be the number of times that the human choice is f = (1 and the sampled pair is (ō, o'), and similar for 2 instead of 1. We obtain
N(ō, o')
N(o 01, log pRo (ōv o*)
ž(0) = Σ
N(ō,ō")
31

---

N(ō,0'22) 2)
log pRo (o' (0' -0)
0")
E
CE
P ( 0') piu (
0,6'~D
=C(
Here, CE is the crossentropy between the two binary distributions. Since we assumed that DD gives positive probability to all observation sequences in Ω and since the cross entropy is generally minimized exactly when the second distribution equals the first, the loss function [()) is minimized if and only if Ro gives rise to the same choice probabilities R for all pairs of observation sequences. Theorem D.2 then gives the result.
D.4 Identifiability of Return Functions When Human Observations Are Not Known
Corollary D.4 assumes that the choice probabilities of each observation sequence pair are known to the reward learning algorithm. However, this requires the algorithm to know what the human observed. In some applications, this is a reasonable assumption, e.g. if the human's observations are themselves produced by an algorithm that can feed the observations also back to the learning algorithm. In general, however, the observations happen in the physical world, and are only known probabilistically via the observation kernel Po. The learning system does however have access to the full state sequences that generate the observation sequences. This leads to knowledge of the following choice probabilities for 8, 8" E S:
E pR P (55 (5> 6,0 (.OPE(-1EE)))
(8)
where the observation- based choice probabilities are given as in Equation (2). In other words, the learning algorithm can only infer an aggregate of the observation- based choice probabilities. Again, we can ask question similar to the ones before, extending the investigations in the previous section:
2,z' is known. Additionally, assume that it is known that the human's 's observations governed by Po. and that the human is Boltzmann rational with inverse lemperature parameter | and beliefs B(3) ō), see Equation (8).
Question D.6. Assume the vector of choice probabilities (P" 5"))
Does this data identify the return function G : S- R.?
If the observation based choice probabilities from Equation (2) would be known, then Corollary D.4 would provide the answer this question. Thus, similar to how previously inverted the belief operator B, we are now simply tasked with inverting the expectation over observation sequences. This leads to the following definition:
Definition D.7 (Ungrounding Operator). The ungrounding operators O
R + R$ and O x O
Ruxî &xS are defined by
[O(i]] (5) = E )) [o()),
[(0
=
E 2.6'Pa((335) [C(ā,a")].
Here, U E R is an arbitrary vector, and C E R also an arbitrary vector, where the notation remind of "Choice" since the inputs to O ② O are, in practice, vectors of observation-bass iitrmannn-rationns choice probabilities.
Formally, O O is the Kronecker product of O with itself, but it is not necessary understand this fact to follow the discussion. Ultimately, to be able to recover the observation-basee choice probabilities, what matters that O @ O is injective on whole vectors of these choice probabilities. The injectivity of O is a sufficient condition for this, which explains its usefulness. We show this in the following lemma:
Lemma D.8. O: R R is injective if and only 0 & O R πxu
5x5 is injective.
We excuse the following abuse of notation: these choice probabilities run through the observations of the human and are not the same the choice probabilities from Equation (1).
32

---

Proof. This is a general property of the Kronecker product of linear operator with itself. For completeness, we demonstrate the calculation in our special case. First, assume that O is injective. Assume that (O & (C)) = 0 for some C' E R We need to show C -= 0.
For all pairs of state sequences (5, 8 have
0 [( ()))))
E
[C(ā,
6.'mPot1555)
E
[((,,T))]
3'~Po(0'33
E
(o)
o (C*)) (5)
where (ō) = Es' [C( This means that for all and o, we have
By the injectivity O, we obtain Cÿ for all 8".
(O)
E 3PPP6(018))
[C(ā, o (c:) (c
where CT = C(ō, σ¹). Again, by the injectivity of O, we obtain C = 0 for all o leading to
C = O. That proves the direction from left to right.
To prove the other direction, assume that O not injective. This means there exists 0 * E I such that O(C) = O. Define C' @ C E Rixū by
(C (C@C)(0, = C(ō)C(O").
Then clearly, C C * 0. We are done if we can show that (O @ O) C) establishes that O 8 is also not injective. For any 5, 5" E S, we have
since that
(0 8 O) 8 C3) (3,6")
( (CO,0)))))
8,5' 3.8"-P((1E.""
E
[(0) ((0))C((')]
E
[C(Ø]
E
[C(o")]
(O()) ((())
0-0
0.
This finishes the proof.
We now state and prove the following extension of Corollary D.4: Theorem D.9. Consider the following statements (which each be true false):
1. R S an injective linear operator: ker O = (0).
2. R RR 5x5 an injective linear operator: ker O & {0).
3. & is injective on vectors of observation- based choice probabilities (p (p= (0)00~0)) 6,5"
over the set of return functions G E R.S.
4. The data of bate-based choice probabilities
(pR(S (51)5 3.8'E5 from Equation (8)
determine the data of observation- -asedd choice probabilities Equation (2).
(p" 0'))
from
0,0'88
33

---

Then the following implications hold and 3 does not imply 2:
3 4.
Consequently, any of the conditions 1, or 3 hold, and additionally any of the conditions (i), (ii) determine the return function G on or from Corollary D.4, then the data (3= 5")) 2,
sequences 5 E S up to constant independent of s.
Proof. That and 2 are equivalent was shown in Lemma D.8. That 2 implies 3 is clear. To prove that 3 implies 4, simply put both sets of choice probabilities into vector. Then Equation (8) and Definition D.7 show the following equality of vectors in Rxx;
Ppn (P"()>))) 3,3" (080) ((P*((>))))) 3,8'
The injectivity of O & O on such inputs ensures that the observation- -bated choice probabilities can be recovered using this equation.
We now show that (3) does imply (2). Again, we use the simple MDP from Equation (7), but this time with a different observation kernel. Namely, we choose
Po(ola) aa) = =Pooo
aa) = Po 1b) 1,
where ola)" * ola) and o(a) + o()) + ola)', This results two possible observation sequences: ol*) (b)) and ala)' (b)) Thus, R is two-dimensional, whereas R is only one-dimensional Con- sequently, O R R cannot be injective, so ker O {0), (2) does not hold since (1) and (2) are equivalent. However, (3) still holds: Since there is only one state sequence, Equation (2) shows that the only vector of choice probabilities has 1/2 in all its entries, irrespective of the return function G. Thus, O @ O has only one input of observation- based choice probabilities, and is thus automatically injective on its inputs.
The final result of identifiability of the return function G follows using Corollary D.4.
D.5 Simple Special Cases: Full Observability, Deterministic Por and Noisy Pō
In this section, we analyze three simple special cases of the general theory.
Theorem 3.9 (together with Lemma B.3) from Skalse et al. [2023], reproduced corollary below, a special case of our theorem:
Corollary D.10 (Skalse et [2023]). Assume the human directly observes the true sequences, and the choice probabilities are given by
P 5"> ( G(5"))
This data determines the return function independent 5.
= ((R) on state sequences ø E Š up to a constant
Proof. We can embed this case into the one of Theorem D.9 by defining the observation kernel Pa(s" 8) 65(5") (i.e., the correct sequence is deterministically observed) and defining the human's 's belief as B(x" 8) = (5(5") (i.e., the human knows that the observation reflects the true sequence). This shows that P( > 5") is of the form of Equation (8). The result follows from Theorem D.9: the operators O and B are the identity in this case, due to the defining property of the Kronecker delta, and they are injective.
The following proposition shows that Corollary D. 1.10 is essentially the only example of deterministic observation kernel Po for which
injective. Note, however, that in some situations, we can have im I r ker B = {0} even if B is not injective, see Example D.32.
Proposition D.11. Assume Pg. the observation kernel on the level of sequences, deterministic and not injective, Then O is automatically injective. However, B is not injective.
34

---

Proof. To show that O is injective, assume E R is such that O(v) 0. Then for all 5 E S, we get
0 = (z)
E (( = v (3))).
Since S f2 is by definition surjective, we obtain
8 + Q is by definition surjective, and here assumed be non-injective, which implies that S has a higher cardinality than Ω. Thus, B R + +R cannot be injective.
In the following, we analyze a simple case that guarantees identifiability. It requires that the observation kernel "well-behaved" that the human is a Bayesian reasoner with any prior (5)) that supports every state sequence J'E S. Definition D.12 (Noise in the Observation Kernel). Then we say that there is noise in the observation kernel Po S + ((2) if = Si and if O is an injective linear operator.
form where the observations are simply "noisy states", and
Proposition D.13. Assume that S = á. Furthermore, assume that B(5 o) is given by the posterior with likelihood P (0 8) and any prior B(3) with (Ø) > 0 Ofror all S. Then there is noise the observation kernel if and only if B is injective.
Proof. Assume O is injective. To show that B is injective, assume there is G" E R with B(G)) O.
Then for all ō E ② we have
0 = [B) = E 2)(()a)
[')())
Σ B( ()) B
Po 8) (B(5) (5))
=[OT(BG (O).
Here, O" is the transpose of O and B G' is the componentwise product of the prior B with the return function G'. Since is injective and thus invertible, O is as well. Thus, O G which implies G" 0 since the prior gives positive probability to all state sequences. Thus, B is injective.
For the other direction, assume B is injective. To show that O is injective, let u E R be any vector with ((v) O. We do similar computation as above: for all 5 R have
[O(') 3-Polo
E
[ot6 Σ (g(ā15)ul) & B( (Po(o)
Bd = =[B" (P8 (3).
Here, BT is the transpose B, Pōj is the denominator Bayes rule, and Po ② the vector with components Po(ō) v(ō). From the injectivity and thus invertibility of B, it follows that B is invertible well, and so Pg O v which implies
Thus, O injective.
Corollary D.14. When there is noise in the observation kernel and the human is Bayesian reasoner with some prior B such that ( > 0 for all E S, then the return function is identifiable from choice probabilities of state sequences even if the learning system does not know the human' observations.
Proof. This follows from the injectivity of O, the injectivity of B that proved in Proposition D.13, and Theorem D.9.
Remark D.15. We mention the following caveat: intuitively, one could think that O (and thus B, by Proposition D. D.13) will be injective if every & is identifiable from infinitely many i.i.d. samples from Polō 8), A counterexample is the following:
1/2
1/4
1/4
0
1/4
1/2
1/4
3/8
3/8
1/4
35

---

In this case, the rows are linearty dependent with coefficients 1/2, and -1 Consequently. O and B are not injective, and so if this observation kernel comes from a multi-armed bandit with three states, then Corollary D.4 shows that the return function is not identifiable.
Nevertheless, the distributions Po(- 8) (given by the rows) all differ from each other, and infinitely many i.idd. samples identify the state sequence S.
D.6 Robustness of Return Function Identifiability under Belief Misspecification
We now again look at the case where the observations that the human observes are known to the reward learning system, as Section D.2. Furthermore, we assume that B R + R is such that ker B Fim I {0). In this case, we can apply Corollary D.4 and identify the true return function G from B(G), which, turn, can identified up to an additive constant from the observation-base choice probabilities with the argument as for Proposition 3.1.
In this section, we investigate what happens when the human belief model is slightly misspecified. In other words: the learning system uses perturbed matrix B == B + A with some small perturbation A. How much will the inferred return function deviate from the truth? To answer this, we first need to outline some norm theory of linear operators.
D.6.1 Some Norm Theory for Linear Operators
In this section, let V, W be two each have inner products ( and there are linear isomorphisms V = R W ~ R such that the inner products in V and W correspond to the standard scalar products in R and RT" The reason that we don't ton't directly work with R and R" itself is that will later apply the analysis to the
inner product-spaces. other words, V and W
that = im r C R Let in this whole section A - W be a linear operator and A : V + W be a perturbance, that A A A is perturbed version of A.
The inner products give rise to a norm on and W defined by
||||| = / (u, v), ||l||| = V(w,w), w).
As is well known, for each linear operator adjoint (generalizing the notion of a transpose) A T W + V such that for all E and W, we have
- W there exists a unique, budis-independent
(A u, w)
U, A w)
Let us recall the following fact that often used in linear regression:
Lemma D.16. Assume A V + W is injective. Then A
V + V is invertible and
(AT A)- AT is a left inverse of A.
Proof.
To show that T A is invertible, only need to show that is injective. Thus, let x E V.
Then
x (I ArA"xx)== = (Az,Ax) = A x1|2 00
where the last step followed from the injectivity of A. Thus, AT A + 0, and so A A is injective, and thus invertible. Consequently, (A" AA- AT is well-defined operator. That it is the left inverse of A is clear.
Definition D.17 (Operator Norm), The norm of an operator A V -> W is given by
|| max 2. ==|=1 Ax]l
It has the following well-known properties, where A, B and C are matrices of compatible sizes:
A + B S
||CA|| < CII | A
To study how perturbance A (and thus A A) transfers into a perturbance of (A A) we will use the following theorem:
36

---

Theorem D.18 (El Ghaoui [2002]). Let B : V + V be an invertible operator. Let < B |_ A is invertible and we have
Let A + V be any operator with
S p. Then
p B
B A
-1
-
B
|
P
Proof. See El Ghaoui [2002], Section and particular Equation 7.2. Note that reference defines || A || to be the largest singular value of A; by the well-known min-max theorem, this is equivalent to Definition D.17.
We will apply this theorem to A A, which raises the question about the size of the perturbance in A A for given perturbance in A. This is clarified in the following lemma. Before stating it, for given perturbance p. define
P(A) p (2 | p). which depends on A and p. Also, recall that for given perturbance A, we define AA = A + A.
We obtain:
Lemma D.19. Assume that A S Then
A
A
S(A)
A
Proof. We have
A AS
A A = (A + A) (A+ A A'
A" A + A A A A
$ A |I A + A || + A
sp. (2 A | +p)
((A).
To be able to apply Theorem D.18 to A A, we need to make sure that P(A) is bounded above by
(A A) -1- The next lemma clarifies what condition p needs to satisfy for ((A) to obey that bound. For this, define
7A) -1 V (A A)-11
which only depends on A.
Lemma D.20. Assume p < TAA). Then
?(A) < (A -)-
Proof. Note that = r(A) is the positive solution to the following quadratic equation in the indeterminate p:
p 2 A ||
= -(A) (A A) ----I---- = 0.
Since this a convex parabola, we get the inequality 7(A) [| A A whenever we have 0 < p < 7(A), which shows the result.
Finally, we put it all together to obtain a bound on the perturbance of ( (A" A)
AT. For this, set
7(A) ( (A" A) 1
(I11I++)) + +AT A) - P. (10)
C(A. p)
((AAA)=11 A) -(AA)
We obtain:
37

---

Proposition D.21. Assume
A
< Then A A is invertible, and we have
[AXAs] A (A- AT S((A
Proof. The invertibility of A A follows from Theorem D.18, Lemma D.19 and Lemma D.20.
We get
A AA -1 A (A"A) |
(A A -1 - (A*A)
A +X+(AA) (AX - AT)
(AX AA - (A |
||
A A)
|A|
((A) A A)
+ (AA++))+|(A77))) -1
(A* A)
-1 ((A)
-CAA p).
In the second-to-last step, we used Theorem D.18.
The constant C(A, p). defined in Equation (10), has fairly complicated form. In the following proposition, we find an easier-to-stude upper bound in a special case:
Proposition D.22. Assume that p || and p -11 A Il Then we have
A
1/2 (| ! 2
C(A, S (A (A A) 12 |
A) (IA
Proof. The second assumption gives, as in the proof of Lemma D.20, that ((A) S 1/2 [[A A) -)- Together with p S || |, the result follows.
D.6.2 Application to Bounds the Error of the Return Function
We now apply the results from the preceding section to our case. Define r(B) im - R as the restriction of the belief operator B to im I. Assume that ker B 3m {0), which is, according to Corollary D.4, a sufficient condition for identifiability. Note that this condition means that r(() is injective. Thus, Lemma 1. ensures that r(B))r((B) invertible and that (r(B)Fr(B)) -1 "r(B)" a left inverse of r(B).
Consequently, from the equation
we obtain
G = (= ((B)
This is the concrete formula with which G' can be identified from B(G). When perturbing B, this leads a corresponding perturbance in (r(B)"r(B)) "r(B)" whose size influences the maximal error in the inference of G. This, in turn, influences the size of the error in Jc. the policy evaluation function, where
Ja( = E
[G(3].
We obtain:
Note the factor 1/2 compared to the definition of r(A) in Equation (9).
38

---

Theorem D.23. Let G be the true reward function, B the belief operator corresponding to the human's 's true belief model ( o), and ( be the resulting observation- baseed return function.
Assume that ker B n im F {0}. so that r(B) r is invertible. Let A R perturbation satisfying
R be
S p. where p satisfies the following two properties:
s [||[B]]
p S - [II[
+ + 1/22- (/2-I(((B))r(BB))
Let B& = B + be the misspecified belief operator: The first claim is that r r(Ba) (BA))i(BB]) is invertible under these conditions.
Now,
assume that the learning system infers the return function G = such that r(Ba)'
(r(Ba)"r(BA)
Then there is a polynomial Q(X,Y) of degree five
- s |G|| (II
Thus, for all policies n, we obtain
Ja(n)- - Ja(a) s || ((I(((((
In particular, for sufficiently small perturbances p. the error in the inferred policy evaluation fuuntioionnuunntionn Jg becomes arbitrarily small.
Proof. That r (Ba)Tr(Ba) ) is invertible follows immediately from Proposition D.21 by using that [|r(A)] I||
A and that r(Ba) = ((B)(()) together with the second bound on (which implies the assumed bound in Proposition D.21).
S ||ll
have
JØ(+) - Ja(T))
[G - (5)
(x)
E
[ (8)
P
max
(G-G)(5)
EE
< || - Gil
[II(B))(((B)) "(BA)
I( ( -((()(((BB [(B)T (I()) B(G)
([()BA))((A))) (r(B)(rBBB)) (B))
(B) | B(G)
C(r(B), [I[
s C(r(B). p)
In the second to last step, we used Proposition D.21. By Proposition D.22, we can define the polynomial Q(X,Y) by
12XY =XY
+1].
which is of degree five.
The last claim follows from limp-op=00
Remark D.24. In the case of a square matrix B that is injective, we can apply Theorem D.18 directly to B (which is now invertible) and obiain the following simplification of Theorem D.23 for the
case that A $ <ps B ||:
(J((T) Ja(n)) S p. 2 |[G 11
The polynomial is then only of degree
'Note that there not necessarily & with r r(Ba) B(G since r(Ba) is not always surjective. Nevertheless, &
-1 "r(Ba) (((B) is the best attempt at solution in the sense that (BB) then minimizes the Euclidean distance (G)).
39

---

D.7 Preliminary Characterizations of the Ambiguity
Recall the sequence of functions
R
R
R
In this section, we clarify im I and ker B in special cases, as their intersection is the crucial ambiguity in Theorem D.2.
The following proposition shows that for deterministic Pa and a rational human, ker B decomposes into hyperplanes defined by normal vectors of probabilities of sequences mapping to the same observation sequence:
Proposition D.25. Assume the human reasons as in Section D.I. Assume Po deterministic. Let ((5) be the distribution of sequences under the human's 's belief over the policy, given by (3) J ((T')P"( (3) for some policy prior B(x²). For each ō. let Bg == [B(3)] 0(ā==
RiSES1 ((3)=ō) be the vector of probabilities of sequences that are observed as ō.
Let G' be return function. For each E á define the restriction G'z E R(8=81Ó())=)) by Gz((3) = (3) for all š E { E Š Õ(š) = o}. Assume that B(3) ō) is the Bayesian posterior. Then G' E ker B if and only if the property
Bø G' = 0
holds for all EE Z.
Proof. For deterministic observation kernel po: by Bayes rule we have
Polō 8) B($)
0(8 Ex Polo 8") B(5))
5o(O(s) B(5)
Σn v. da (0())))) B(5")
0, Ō(5) A0
B(3)
Ö(5) J.
Ez, 0: B3)
Thus, for any return function G' and any observation sequence o, we have
(B) B-((3))
E [G)(5)
Σ B(3 ())
B(s)
Σ
G'(5)
Σx 0( B(3")
3: Ō(3)=
Σ B(3")
Σ (3)
3y: Ó(3"=z
& Ó
Thus, we have G E ker B if and only if
Bg G
Σ
8: Ú
for all U. That was to show.
Remark D.26. One can interpret the previous proposition as follows:
As long as Ô is injective, we have {& 5 Ō(3) = o} = 1 for all o, meaning that By and Gy have only one entry. Thus, B3- G = 0 implies G² If that holds for all o, then G' E ker B implies G" = O, meaning B is injective.
40

---

However, as soon as there is an ō with k3 k K
= = 1, the equation B; G' 0 leads to kig free parameters in G G'g can then be chosen freely in the hyperplane of vectors orthogonal to B3 without moving out of the kernel of B.
Another way of writing Proposition D.25 is to write ker B as a direct sum of these hyperplanes perpendicular to Bg:
ker
B
3: (0-
Recall that a return function is called separabe if there exists a reward function R such that [(R) = G.
Before we discuss time-separability more interesting examples, we want to talk about one simple case where all return functions are time-separable. We leave general characterization of im I to future work.
.... of all sequences in S, and a function 6 S from sequences to states such that (x3) E 5 and o () & s for all i k. Then every return fumction is time-sepe sepaaable......
Proposition D.27. Let there be an ordering
Proof. Let G be a return function. Initialize R(s) = 0 for all and inductively update for all i i= 1.
R(0(5(0)) =
Σ
G G(S
Σ
7
t:
I: si M0(
where the inductive definition always uses R as it is defined by that point in time. Once R R(o(5())) (((())) defined, but yet any future values R()(s()))) h > we have
T
T [(R)
= Σ
r
R(s
1=0
x) R((()(0)))
Σ
+
Σ
7' -(s((s)))
t: 6 (Po0(20))
G(
Furthermore, the property ((5()) & 5(0 for all l ensures that changes the reward function for k i do not affect the value of [T()] (51)). This shows r(R) G, and thus G is time- separable.
Corollary D.28. In multi-armed bandit, every return function is time seme-aeparable
Proof. In a multi- bandit, states and sequences are equivalent, and so we can choose ((s) = for every state/sequence The result follows from Proposition D.27.
Alternatively, simply directly notice that multi- bandit, the identity mapping, and so for every return/reward function R, we have I(R) = R.
D.8 Examples Supplementing Section 5
In this whole section, the inverse temperature parameter in the human choice probabilities is given by 3 = 1. We now consider four more mathematical examples of Corollary D.4 and Theorem D.9. In the first example, the ambiguity is so bad that the reward inference can become worse than simply maximizing Jobs as in naive RLHF. In Example D.30, there is simply "noise" in the observations and human's 's belief, the matrices and are injective, and identifiability works, as in Corollary 14. In the third example, the matrix B is not injective and identifiability fails, which is a minimal example showing the limits of our main theorems. In the fourth example, the matrix B not injective, but ker B im I {0). and so identifiability works. This example is interesting in that the identifiability
41

---

simply emerges through different distributions of delay that are caused by the different unobserved events.
In this section, both the linear operators B matrices
RS
RR and O R + RS are considered as
O = (Pálo 8) g,
Sx
B (B(5 σ)) 03 R
Notice that both have swap their indices.
Example D.29. Theorem 5.2 shows that the remaining ambiguity from the human's choice probabil- ities is given by ker B n im T, but it doesn't I explain how to proceed given this ambiguity. Without further inductive biases, some reward functions within the ambiguity of the true reward function can be even worse than simply maximizing Jobe
E.g., consider a multi-ammed bandit with three actions 0, b, C, observation-kernel
O(a)
O(b) # (c)) = c. and reward function R(a) = R(b) < R(c). If the human belief is given by B(a 0) = p == B(b o), then R' = a (p 1. 0) E R (a.b.c) in the ambiguity for all E RR and so R = R+ R' is compatible with the choice probabilities. However, for a & O, we have R(a)) > R(b) and R((a) > R(c), and so optimizing against this reward function leads to a suboptimal policy.
In contrast, maximizing Jobs leads to the correct policy since a, b, and e all obtain their ground truth reward in this example. This generally ruises the question of how to tie-break reward functions in the ambiguity, or how to act conservatively given the uncertainty, in order to consistently improve upon the setting in Section 4.1.
Example D.30. This example is a special case of Corollary D.14. Consider a auuti--rmed bandit with two actions (which are automatically also states and sequences) a and b. In this case, the reward function and return function is the same.
We assume there be possible observations d(a), (0()) and the observation kernel be non- deterministic, with probabilities
2/3, if Po(o (
1/33, else.
If we assume the fuman.form. Bayesian posterior beliefs as in Section D.1 and to have policy prior (xx') such that ((a)= = J= B(a)B(rrt )dπ 1/2 and B(b) 1/2, then it is easy show that the human's 's belief is the "reversed" observation kernel:
B(j 10(0) P)( (5).
We obtaîn
2/3 1/3
2
1 2
O= B
1/3 2/3)
3
1
These matrices are injective since they are invertible:
2 -1
-
o --
-1 2
More generally, even if the human does not form.fully rational posterior beliefs, it is easy to imagine that the matrix can end up being invertible. Thus, Corollary D.4 guarantees that the reward function can be inferred up to additive constant from the choice probabilities of observations, and Theorem D.9 shows that this even works when the learning system does not know what the human observed.
In the rest of this example, we explicitly walk the reader through the process of how the reward function can be inferred, in the general case that the observations are not known. In the process, we essentially recreate the proof of the theorems for this special case. For this aim, we first want to compute the choice probabilities p > i) that the learning system has access to in the limit of
infinite data. We assume that the reward function is given by R(a) = -1 and R(b) = 2. We compute:
1 (2
1
(
B(R) = 3
2)
=
42

---

In other words, we have Bssooo)) [R(s)] [R())
0 and B(j [R(s]] = 1. From this, we can
compute the observation- based choice probabilities Pawso = o ( - ((R)(ol)))) see Equation (2), and obtain:
Palrooo PoNaN 2' Poinsoos = 1
P Pamaon e 1+e
We can now determine the final choice probabilities Púj == Pi > j) again by matrix product, with the indices ordered lexicographically, see Equation (8). Here, O & O is the Kronecker product of the matrix O with itself:
4
2
2
1
1/2
1/2 1/3 (2 ec)
2
1/(1+()
P (0 8 0)
2
4
2
c1(1+e) +e)
1/3 (1 + 2 1/2
2
4
1/2
For example, the second entry in is Pab = PR > b)
2+c
This is the likelihood that,
3-(1+e)'
for ground-truit truth actions a, b, the human will prefer a after only receiving observations of*) or (( according O and following coltzman-rational policy based the belief of the real action, see Equation (8).
Over time, the learning system will be able to estimate these probabilities based on repeated human choices, assuming all state-pus mate-pairs are sampled infinitely often. The question of identifiability whether the original reward function R can be inferred from that data, given that the leurning system knows O and assume that the learning system doesn't 't a priori know or any of the intermediate steps in the computation. First, P can be inferred by inverting O 080:
4
-2
-2
1
1/2 1/3 (2 e)/(111 +e)
1/2 1/(++e)
(0
-2
2
2
4
-2
1/3 (1 2e) 2L)(( 1/2
c1(1+e)
1
-
-2
4
1/2
The learning system wants to use this to infer B(R) (for the later-to-be inferred reward function R that may differ from the true reward function R) and uses the equation
B (o))))
exp
Polssote
exp ((R)(ol()) + exp ((R)
which can be rearranged to
((A)) = -log Palnrotos
+ (( = log 1/(1 +e) e/(1 +e)
(() =B()(
1 Painron
This relation is all which can be inferred about B (ol(()) and B (oft) the precise value cannot be determined and B ()) is free parameter. One check that for B B(R)(o())) this coincides with the true value BRR). Finally, one can invert to infer R from this:
B B(R)
-1))) -1 2
2 -1)
(B(1)
1) (o()))
(ol) -2
1 ((+) (O))
(o
((
(B(()(((()) (( -1
R+
Thus, the inferred and true reward functions differ maximally by constant, as predicted in Theo- rem D.9.
43

---

In the following example, we work out a case where the reward function is so ambiguous that any policy is optimal to some reward function consistent with the human feedback:
Example D.31. Consider a multi-armed bandit with exactly three actions/states a, b, c. We assume deterministic observation kernel with O(a) O(c) + O(b) = b. Assume the human has some arbitrary beliefs (a B(c = =1 B(a o), and can identify b: B(b b) = 1. Then if the human makes observation comparisons with a Boltzman-rational policy, as in Theorem D.2, the resulting reward function is so ambiguous that some reward functions consistent with the feedback place the highest value on action o, matter the true reward function R. Thus, even the true reward function R regards a as the worst action, can result from the reward learning and subsequent policy optimization process.
Proof. The matrix B Rlabbc) + Rfo,bt is given by
B 0) B(e B)clo) o
B
Its kernel is given by reward functions R' with (b) = and (c(c) = Bialo) Ra),), with (a(a)
B(clo)
free parameter. Theorem D.2 shows that, to an additive constant, the reward functions consistent
with the feedback of observation comparisons are given by it = R + R' for any N8' E ker B. Thus, whenever the free parameter (a) satisfies R' Ra(a) R(b) R(a) and R (a) B(c
o) (R(c) R(a)), we obtain R(a) > R(b) and R(a) > R(c). showing the claim.
We now investigate another example where B is not injective, and yet, identifiability works because B r # {0}. We saw such cases already Example E.6, but include this additional example since it shows a conceptually interesting case: two different states lead to the exact same observations, but can be disambiguated since they lead different amounts of delay until a more informative observation is made again.
Example D.32. In this example, assume that the human knows the policy a that generates the state sequences (corresponding to a policy prior ((T') = (z') concentrated on T), which together with knowledge of the transition dynamics of the environment determines the true state transition probabilities T (s' {) = E.EA T(s' T(s' 8, 8.a) x(a s). We consider an environment with three states 8, s', s" and the following transition dynamics T where p + 1/2 is a probability:
1/3
1/3
1/3
1-p P
s'
s"
21-p
We assume that Po(s) 1. Furthermore, we assume deterministic observations and s O(s) # Oss)) = O(s") = O.
Assume the time horizon T is 3, ie., there are timesteps 0, 2,2,3. Assume that the human forms the belief over the true state sequence by Bayesian posterior updates as in Section D. In this case, ker B + (0) by Proposition D.I.. However, we will show that ker(B r) {0)- makes Boltzmann-ration
the human
comparisons of observation sequences, then this implies the identifiability of the return function up to an additive constant by Corollary D.4.5
Thus, let R' E kcrr i.e. B (T(R')) = 0 for every observation sequence ō. For
= 5558 being the observation sequence that only consists of state s, this implies R²'(s) = Consequently, for general observation sequences o, we have:
3
3 (s)))s) ot - R'(s')+ B-B( 1=
0 = B (T(R)) B
(s) .2
R'(s").
t=)
'We assume that the learning system knows what the human observes, which is valid since Po is deterministic. Alternatvely, one can argue with Proposition D. 1.1110 that O is automatically injective, meaning one can apply Theorem D.9.
44

---

Now we specialize this equation to the two observation sequences ot soss and o. soos.
We start by considering d. This is consistent with the two state sequences g()),(+)) ss' 88 and g()).(*)) = sssss. ss. We have posterior probabilities
B B (d4) =1 =1- -p, B(5ti)((")) 1041)= =p.
and therefore
0= 0= [ (0(1))
(011) = -1 (1-p) p)p . R'(s') + p. Pp.".
and
R²'s') p p-1 (2
(1I)
Similarly, of2) consistent with the sequences g(2))(6') 8s' 5 and 5(2), 88" 8" They have posterior probabilities
B ((32)
1 B(x(2)(6)) d22) = -2'
leading to
0 0= (7 (7) (2'))) + 1 1/2 (7+ R((d").
Together with Equation (11), we obtain
R²(s") = -R' 1-p
p Rt' R's"),
which implies (s(")) because p + 2. and thus also (s's') O. Overall, have showed R' = 0, and B o I is injective. This means that reward functions are identifiable in this example up to an additive constant, see Corollary D.4.
E Issues of Naively Applying RLHF under Partial Observability
In this section, we study the naive application of RLHF under partial observability. Thus, most of it takes a step back from the general theory of appropriately modeled partial observability in RLHF Later, will analyze examples where we also apply the general theory, which is why this appendix section comes second.
In Section E. we first briefly explain what happens when the learning system incorrectly assumes that the human observes the full environment state. We show that as a consequence, the system is incentivized to infer what we call the observation return function Gohs, which evaluates state sequence based on the human's 's belief of the state sequence given the human's 's observations. In the policy optimization process, the policy is then selected to maximize Jobs expectation over Gobs- In the interlude in Section E.2, we then briefly analyze the unrealistic case that the human, when evaluating policy æ, fully knows the complete specification of that policy and all of the environment and engages in rational Bayesian reasoning; in this case, Jobs = J is the true policy evaluation function.
Realistically, however, maximizing Jois can lead to failure modes. In Appendix E.3 we prove that a suboptimal policy that is optimal according to Jabs causes deceptive inflation, overjustification. or both. Appendix C.3, we expand on the analysis of the main examples in the main paper. Finally, in Section F.4, we study further concrete examples where maximizing Jobe reveals deceptive and overjustifying behavior by the resulting policy.
E.1 Optimal Policies under RLHF with Deterministic Partial Observations Maximize Jobs
Assume that Po is deterministic and that the human makes Boltzmann sequence comparisons between observation sequences. The true choice probabilities are then given by (See Equations (2) and (8)):
pR P7 5*) = (B(()))(()6))) - (B (B)G))
(12)
Now, assume that the learning system does not model the situation correctly. In particular, we assume:
45

---

The system is not aware that the human only observes observation sequences ((5) instead of the full state sequences.
The system does not model that the human's 's return function is time-separable ie., comes from a reward function R over environment states.
The learning system then thinks that there is return function G probabilities are given by the following faulty formula:
R such that the choice
p (8 + 5*) = (B(G(8)
Now, assume that the learning system has access to the choice probabilities and wants to infer G. Inverting the sigmoid function and then plugging in the true choice probabilities from Equation (12), we obtain:
log ( 8")
G(5) 3 p + G(5")
(B(((()))((BBB))))BBB =(B (B-G) CC(5))
B ((B)) - (B (B-G)(((5))) +G'5")
Here, C(5") is some quantity that does not depend on 5. Now, fix g" as a reference sequence. Then for varying s, ((5")) is simply an additive constant. Consequently, up to an additive constant, this determines the return function that the learning system is incentivized to infer. We call it the observation return function since it the return function based on the human's 's observations:
Gobs (3) = (B- (O(5).
This return function is not necessarily time-separable, but we assume that time-separability is not modeled correctly by the learning system. Now, define the resulting policy evaluation function Johs by
(obs(a)
E
(5)
8~P
This is the policy evaluation function that would be optimized if the learning system erroneously inferred the return function Gobs-
E.2
Interlude: When the Human Knows the Policy and is Bayesian Reasoner, then Jobe =
In this section, we briefly consider what would happen if in Jobs the human's belief B would make use of the true policy and be rational Bayesian posterior as in Section D.1. We will show that under these conditions, we have Jobe J. Since these are unrealistic assumptions, no other section depends on this result.
For the analysis, drop the assumption that the observation sequence kernel Pa is deterministic, and assume that Jabs is given as follows:
Jobe(TT =
E Er
E 8~ Po
E (3'10
[G(S")]
(13)
In this formula, ( σ) Formally, this the posterior of the joint distribution B(5, o m) that is given by the following hidden Markov model:
B( o. T) with B being the joint distribution from Section D.1.
80
81
82
83
Po
Po
Po
Po
(14)
00
"Note that the case of non dendettermiiiticc observation kernels and choice probabilities given as in Equa- tion (8). this argument does not work since the logarithm cannot be swapped with the outer expectation of the choice probabilities.
46

---

= EBEA T(s' s.a) (a s). is sampled according to the known initial distribution Po(so). The human's posterior 8 obtain:
Here, (= 1 s)==
(5" o) is then the true posterior in this HMM. We
Proposition E.1. Let π be policy that is known to the human. Then Jobs (T) ((x).
Proof. By Equation (13), we have
Jnba (T)
E
E
E
(G("")
New pe (3)
How (
(5-10)
PaloT 8) Σ
(5 |
8
(2)
Σ B" (3" σ) Σ P
5)
G(3²)
8'
(8)
Σ
Σ (3 () G(3*)
3'
Σ
Σ
PT (8)) Po 8*) G(5*)
(5)
Σ
P" (3"))((3")
8'
(
Σ
(T(())(())
(TT).
In step (1), we wrote the expectations out in terms of sums. In step (2), we reordered them. In step (3), observed that the inner sum over 5 evaluates the marginal distribution Bd (Ø) of the observation sequence o in the HMM in Equation (13). In step (4), we used Bayes rule in the inner sum. This is possible since (5" ō) is the true posterior when क is known. In step (5), we pull (*(5") out and notice that the remaining inner sum evaluates to 1. Step (6) is relabeling and step (7) the definition of the true policy evaluation function J.
E.3 Proof of Theorem 4.5
We first prove the following lemma.
Lemma E.2. Let and Ved two policies. If J(x) < Hars) and Jors((T) JohnsTTre) then relative to ref a must exhibit deceptive inflation, overjustification, or both.
Proof. We start by establishing a quantitative relationship between the average overestimation and
underestimation errors E and E as defined in Definition 4.2, the true policy evaluation function
J, and the observation evaluation function Jobs defined in Equation (4). Define A : S - R. by
s = Gobs G(3), where Gobs as defined Equation (3). Consider the quantity
(T - (T) max (0, ((3) max (0,-)(5))
(8) > O, then the first term is A A(8) and the second one If A(5) < O, then the first term zero and the second one is (8)) If (55) = 0, then both terms are zero. In all cases the right-hand side equal (z). Unpacking the definition of again, we have that for 8,
E E (5) Gobs G(5).
(15)
For any policy n, if we take the expectation of both sides of this equation over the on-policy distribution admitted by n, pa we get
E (TT - E (T) Jobs J (TT).
(16)
47

---

We now prove the lemma. Let and Thet be two policies, and assume that J(T) < (neet and
Johs (TT) 2 (Frr) Equivalently, we have (oha(TT) Wibbs(TTee) 0 and ( [
which we combine to state
(Joha(TT= - -ntta -nnba(TTret) -
J(T)) 0.
(17)
Rearranging terms yields
( J(T))
(Joba(Tret) - J ( > 0.
These two differences inside parentheses are equal to the right-hand side of (16) for π and Tef respectively. We substitute the left-hand side of (16) twice obtain
E (x) (0)-E(()) (m))
-(**
(E* (TTret - E (Tree)) > 0.
Rearranging terms again yields
(T)(
(T) E (TTeet) + E (TTet) E' (T) > O.
(18)
EE (Tt) E. (Ttee) > 0 then we have E. (T) > E. (Tret and, by assumption, Jubs( >
Jobs (T By Definition 4.3, this means exhibits deceptive inflation relative to Tra--
If E (Fet) E. (x) > 0 then we have (T)) < E (mret) and, by assumption, J(r) < J (Fret)
By Definition 4.4, this means π exhibits overjustification relative to Teef
At least one of the two differences in parentheses (18) must be positive, otherwise their sum would not be positive. Thus must exhibit deceptive inflation relative to Ter overjustification relative to Tyt or both.
We can now combine earlier results to prove Theorem 4.5, repeated here for convenience:
Theorem F.3. Assume that Po is deterministic. Let Thhs be an optimal policy according to a naive application of RLHF under partial observability, and let n² be an optimal policy according the true objective J. If Ths is not J-optimal, then relative to x." Tobs must exhibit deceptive inflation, overjustification, or both.
Proof. Because Po is deterministic, Tahe must be optimal with respect to Jobe by Proposition 4.1 (proved in Appendix E.1). Thus Jobs (Taba)) 2 Jobe (T*). Since л² J-optimal and The not, (xx)) < (.Toh)) ). By Lemma E.2, relative to π², The must exhibit deceptive inflation, overjustifica- tion, both.
E.4 Further Examples Supplementing Section 4.4
In this section, we present further mathematical examples supplementing those in Section 4.4. We found many of them before finding the examples we discuss in the main paper, and show the same and additional conceptual features with somewhat less polish. We again assume that Po is deterministic.
Example E.4. In the main paper, we have assumed a model where the human obeys Eq. (2) and showed that naive application of RLHF can lead to suboptimal policies, and the specific failure modes of deceptive inflation and overjustification. What if the human makes the choices a different way? Specifically, assume that all we know is that ( > σ') + p > ō) 1. Can the human generally choose these choice probabilities in such a way that RLHF is incentivized to infer a reward function whose optimal policies are also optimal for R.? The answer is no.
Take the following example:
In this example, there fixed start state and three actions a, b, e that also serve as the final states. The time horizon is T = 1, so the only state sequences are sa, sb, sc, Assume T(a s, a, 1.
48

---

T(b - i, - s.c) - i.e. selecting action sometimes leads to state Also, assume O(a) 0(b) O(c) and R(a) R(b) R(c).
Since b and e. have the same observation o the human choice probabilities do not make a difference between them, and so RLHF is incentivized to infer reward function with R(b) = R(c) = R(o). If R(o)= > R(()), then the policy optimal under R will produce action since this deterministically leads to observation o, whereas с does not. If R(o) < R(aa,, then the policy optimal under R will produce action a. In both cases, the resulting policy suboptimal compared to TT, which deterministically chooses action c.
In the coming examples, it will also be useful to look at the misleadingness of state sequences:
Definition E.5 (Misleadingness). Let E SES be a state sequence. Then misleadingness is defined by
E M(3) Gohs (8) G(8) B(=B((1((G))
(G(5")
We call a state sequence positively misleading if (() > 0, which means the sequence appears better than it is, and negatively misleading if (3) < 0. The misleadingness vector is given by M E R
Note that the misleadingness related to E+ and E defined in Definition 4.2: M(5) > then M(5) E (5), and if M(x) < O then M(5)
(5).
Example E.6. In this example, assume the human is Bayesian reasoner in Section D.I. Consider the MDP that suggestively depicted as follows:
c
The MDP has states S = {a. b, c} and actions A = (b,c)........ c}. The transition kernel is given by T(c meaning that the action determines whether to transition from a to or c. All other transitions are deterministic and do not depend on the action, as depicted. We assume an initial state distribution Po over states with probabilities pa Po(a), Pb Po Pe Po(c). The true reward function R (a.be) and discount factor y E (0.1) are, for now, kept arbitrary. The time horizon is T = 2, meaning we have four possible state sequences acc, abc, bcc, ccc.
a, c) = 1 and (b a.
Furthermore, assume that O(a) O(b) O(c) i.e., e is observed and and ambiguous.
Finally, assume that the human has policy prior B()), where X = Tx(C a) is the likelihood that the policy chooses action when in state a, which a parameter that determines the entire policy.
We claim the following:
1. If + 7 Ex-Box|A] Pa. then ker Boim = (0), so there is no return function ambiguity under appropriately modeled partially observable RLHF, see Corollary D.4.
2. There are true reward functions R for which optimizing Johs leads to suboptimal policy according to the true policy evaluation function J, case of misalignment. Thus, naive application of RLHF under purtial observability fails, see Section 4.1.
3. The failure modes are related to hiding negative information (deception) and purposefully revealing information while incuring a loss (overjustifying behavior).
Proof. Write p = [bcc occ). the human's posterior probability of state sequence bcc for observa- tion sequence occ. We have p = B(ace) occ).
Consider the linear operators I IR (a.bc) R. (oc,, and B R bec,,,,,,,
R pocc defined in the main paper. When ordering the states, state sequences, and observation
49

---

sequences we just wrote down, we obtain
72
0
0
1
7 x2
0
7
r
7 7 7+72
B =
0
1-p
Bor- =
1 1-p
7
1 0
1
By Corollary D.4, if B T is injective, then there is no reward function ambiguity. Clearly, this is the case if and only if p # Y p). From Bayes rule, we have
B(bec)
((ace)
p
B(acc) + B(bec)
p B(ace) + B(bec)
So the condition for injectivity holds if and only if
B(bcc) # 7 B(acc).
Now, notice
B(boc) J B(A) B(boe) A)dA J B(A) PPdd Pd
and
B(ace) J ((A)B(acc) A)dX / B()) Paa Pa AdA Pa
E A~B(A)
[X]
This shows the first result.
For the second statement, explicitly compute Johs up to affine transformation, which does not change the policy ordering. Let R be the true reward function, G = ((f) the corresponding return function, and B (G) the resulting return function the level of observations. For simplicity, assume R(c) = 0, which can always be achieved by adding a constant. We have:
Jobs (A)
E (P))
((G)))
(abe)) (OG)) + pd (bee) ((G)( + (ccc) ((G)( + Pa(ac)) (OG)
Pa (1- G((ab)) + pb (OG) Pe G(ece) + Pa A (OG))
A. ((G))) G(abc)
have
G(abc) = R(a) +*6(6, (OG) = -1-p) -G(acc) +p- G(bec) = (1 p) R(aa) -p R(b).
Thus, the condition (oce) G(abc) is equivalent
R((a) < P-Y R(b).
p
Thus, have
1, if R(a)
< P-x R(b), p
arg max Jabsa (X) xe|o.1]
0, else.
Now consider the case R(b) > 0. In this case, = gives rise to the optimal policy according to G since going to gives extra reward that one misses when going to e directly. However, when R(a) < 0, then Jubs selects for À 1. Intuitively, the policy tries to "hide that the episode started in a" by going directly to c, which leads to ambiguity between acc and bcc. This is a case of deceptive inflation in Theorem 4.5.
gives rise to the optimal policy according to G. However, when R(a) > 0, then Johs selects for À = O. Intuitively, the policy tries to "reveal that the episode started with a" by going to b, which is positive information to the human, but negative from the perspective of optimizing G. As in Theorem 4.5, we that this is case of overjustification D
Now, consider case R(b) < 0. In this case,
50

---

Example E.7. In this example, we consider an MDP that's similar to multi-armer bandit with four states/actions a, e, d and observation kernel O(a) O(b) + O(c) O(d). Formally, we can imagine that it is given by the MDP
d
with R(s)=00 and time-horizon of = 1. In this example, we reveal that misleadingness and non-optimality (according to the true reward R, or J) are in principle orthogonal concepts. We consider the following four example cases. In each one, we vary some environment parameters and then determine Caber the action that results from optimizing Jubs (corresponding to naive application of RLHF under partial observabiliry, see Section 4.1), its misleadingness M(a (see Definition E.5), and the action a" that would result from optimizing for the optimal action. For simplicity, action results eventually (out of the action taken and potentially a deviation defined by e, see below) is taken before making an observation, i.e. B(a)= = B(b) = B(e)= ((d)==
If dobs = a", then Jobs selects can imagine that the human has uniform prior over what
(a) Assume R(a)= > R(e)) > R(d)= > R(b). Also assume that action leads with probability e > to state b, whereas all other actions lead deterministically to the specified state. Then 0ubs
Míc) < and a* a.
(b) Assume R(d)= > R(a)= > R(c) > R(b). Again, assume there is a small probability e > that action d leads state b. Then daas M(c) > O, and a" = d or a" = depending the size of
(c) Assume R(a) > R(b) > R(c) > R(d). Additionally, assume that there is a large probability > 0 that action a leads to state d, whereas all other actions lead to what's 's specified. If is large enough, then a² = b. Additionally, we have d'us= = and M(b) > O.
(d) Assume R(a)= > R(b)= > R(c) R(d). Also, assume some probability >> that action b leads to state d, whereas all other actions lead deterministically to what's 's specified. Then aobs M(a) < 0, and a.
Overall, we notice:
Example (a) shows a high regret and negative misleadingness of dobs c. The action is better then it seems, but action a would be better still but cannot be selected because it can confused with the very bad action b.
Example (b) shows a high regret and high misleadingness of aotss c. The action is worse than it seems and also not optimal.
Example (c) shows zero regret and high misleadingness of a = b. The action is worse than seems because it can be confused with a, but it still the optimal action because can turn into d.
Example (d) shows zero regret negative misleadingness of d' = 0. The action is chosen even though it seems worse than and is also optimal.
Thus, we showed all combinations of regret and misleadingness of the action optimized for under Jobs-
We can also notice the following: Examples (a) and (b) only differ in the placement of R(d). In particular, the reason that on c is structurally the same in both, but the misleadingness changes. This indicutes that misleadingness not on its own contributing to what Jobe optimizes for.
The following is the smallest example we found with the following properties:
There a unique start state and terminal state.
A naive application of RLHF fails in way that shows deception and overjustification.
Modeling partial observability resolves the problems.
51

---

Example E.8. Consider the following graph:
A
C
B
This depicts an MDP with start state S. terminal state T and possible state sequences STTT, SATT, SACT, SCTT, SCCT,,,BCT,SDL are observed, i.e. O(S)
and no discount, i.e. Y 1. Assume that S, B,C O(B) B, O(C) -C, and that A and T are ambiguous: O(A) O(T)= = X. Then there are five observation sequences SXXX,SXCX, SCX SBCX, SBXX. Assume that the human can identify all observation sequences except SXXX, with belief b B(STTT SXXX) and B(SAT)) SXXX).
Then the return function is identifiable under these conditions when the human's 's belief is correctly modeled. However, for some choices of the true reward function R and transition dynamics of this MDP, we can obtain deceptive or overjustified behavior for naive application of RLHF
Proof. We apply Corollary D.4. We order states, state sequences, and observation sequences as follows:
S = B,,,
5 = STTT, STTT,,ATT, SATT, SCT,,
c = SXXX, SCXX, SBXX.
As can easily be verified, with this ordering the matrices B E R
RS8 are given by:
and r
E
1
0
3
1
0
0
0
0
0
2
0
0
0
1
1
B
0
0
0
1
0
0
r
1
2
0
1
1
0
0
0
0
0
1
1
0
0
2
To show identifiability, we need to show that ker B = {0). Clearly, the kernel of B given
by all return functions in R that are multiples of G' = =(b -bb- 166-1,44 b, 0,0, 0, 0). Assume G' E im meaning there reward function R' E R with P G¹. We need to deduce from this contradiction. The assumption means we obtain the following equations:
ii R'( (1()) 6-
(M) IT(S) + +
(iii) R((A) RCC)+ R'(T)
(iv) {(())++((()++(())=
(v) (()))=))()))++(()+==()))+
(vi) R(S)+
(iii) and (v) together imply (²) x'(A)) (iv) and (vi) together imply R² RC) (v) and
(vi) together imply R'(C') R'(T); together, we have RA) (T) Thus, replacing R' in
(ii) by R' (T) and comparing (i) and (ii). we obtain b = b. contradiction. Overall, this shows ker B Nimm I {0), and thus identifiability of the return function by Corollary D.4.
Now investigate the case of unmodeled partial observability.
For demonstrating overjustification, assume deterministic transition dynamics which every arrow in the diagram can be chosen by the policy. Also, assume R(A) < 0. R(T) 0, R(S) 0.
52

---

R[ O, and R(C) = 0. Then the optimal policy chooses the state sequence STTT. However, this trajectory has low observation value since (STT(T (Bx
(1 - (STTT) which is low since R(AA) & 0. Johs then selects for the suboptimal policies choosing SBTT or SCTT, which is overjustified behavior that makes sure that the human does not think state A was accessed.
- R(B) - R(C) = 0 and For demonstrating deception, assume that R(A) O, R(Z) < O, R(S)
that the transition dynamics are such that when the policy attempts to transition from S to A, it will sometimes transition to B. with all other transitions deterministic. In this case, the optimal behavior attempts to enter state A since this has very high value. Jobs. however, will select for the policy that chooses STTT. This is deceptive behavior.
53

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
This can be verified by reading the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In Section we have a paragraph on limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
54

---

Justification: All theorems come with full set of assumptions, with full proofs in the appendix linked. Sometimes, "background assumptions", like the fact that we study an underlying MDP with an additional observation kernel (o 8), or that the human comes with belief kernel B( o). are omitted in the theorem statements since they apply throughout to the whole paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [NA]
Justification:
The paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- spen-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
55

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [NA]
Justification:
The paper does not include experiments requiring code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines lattps://nipe) cc/pu
ss onPolicy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips. cc/public/geteeecccccccccc cabbiccgguiess
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA]
Justification: The paper does not include experiments.
Guidelines:
The answer means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[NA]
Justification:
The paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
56

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [NA]
Justification:
The paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics tttta://neerre http//ubbic/Bthicuaiieieeeeeeeee
Answer: [Yes]
This research does not involve human subjects, does not make use of data, and does not propose a practical method that could misused or have a negative impact. As such, the paper does not give rise to any ethical concerns.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Ys]]
The last paragraph of the main paper is an impact statement, listing the positive impact we hope to see from our work. As our work is theoretical and does not provide method, no negative impact arises from it.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
57

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
58

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode camdaataees has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
59

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
60


Reviews:
[{"review_id": "Mrv9BbQdjP", "reviewer": "Reviewer_v92A", "summary": "The paper addresses the problem of accounting for humans having only partial observability of their environment when providing feedback. They outline two natural issues that can arise from such partial observability - deceptive inflation and overjustification - and provide examples of both. They then explore what is possible if the human's partial observability is fully known and accounted for, specifically asking how much about the return function can be recovered from the feedback. They show that in many realistic cases, there is an irreducible ambiguity (formalized as a vector space) in determining the return function. Finally, they propose ideas for combating these issues with a small proof-of-concept theorem backing their ideas.", "questions": "1. Are there easy ways to force the ML model to\"expose the underlying state a bit more\" from time to time? A naive example is forcing verbosity in the case of the deceptive inflation example, but of course one needs to design a more general and sustainable fix.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles a new facet of partial observability in human feedback - humans not observing the entire state.\n2. They formalize the intuitive issue of deceptive inflation, and also introduce the counterintuitive but very plausible idea of overjustification.\n3. Their theoretical framework is very clean and allows them to completely characterize the ambiguity in obtaining the return function.\n4. The study of examples in the appendix is very comprehensive.", "weaknesses": "1. While the problems outlined are illustrated using hypothetical examples, I think the paper would greatly benefit from carefully designed experiments that unambiguously demonstrate the existence of such problems in practice.\n2. The paper does not propose any concrete solutions to the problem. While this is great work exploring the problem, I feel that that does not rise to the level of a NeurIPS paper. This would make an excellent workshop paper that can grow into a full conference paper after adding some concrete attempts at solving this problem, either by presenting algorithms with new theoretical guarantees in this context, practical experiments showing a quantifiable improvement in meaninigful metrics associated to the problem, or both.\n3. Minor and easy to rectify: The paper does not discuss other work in RLHF that deals with partial observability and heterogeneity (albeit in a subtly different context), for example [1, 2, 3].\n\nAll in all, this paper is a great start, but needs more work to become a full-fledged conference paper. I am willing to reconsider my score if either 1 or 2 are provided.\n\nRefs:\n\n1. Direct Preference Optimization With Unobserved Preference Heterogeneity. Chidambaram et al, 2024.\n\n2. A Theoretical Framework for Partially Observed Reward-States in RLHF. Kausik et al, 2024.\n\n3. RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation. Park et al, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review!\n\n>The paper would greatly benefit from carefully designed experiments that unambiguously demonstrate the existence of such problems in practice.\n\nOur work is theoretical and we advocate for judging it on a theoretical standard. We explain our viewpoint in more detail in our [global rebuttal](https://openreview.net/forum?id=XcbgkjWSJ7&noteId=dwv3iH9Ccw).\n\n> The paper does not propose any concrete solutions to the problem [...] adding some concrete attempts at solving this problem, either by presenting algorithms with **new theoretical guarantees** [emphasis ours] in this context, practical experiments showing a quantifiable improvement in meaninigful metrics associated to the problem, or both.\n\nWe want to highlight that Section 6 presents exploratory ideas for practical solutions, and in particular, that the discussed Appendix C.3 with Proposition C.5 provides **a first theoretical guarantee**: If the belief matrix B is known explicitly, one can design a differentiable loss function whose minima are feedback-compatible. In particular, this proves that if the ambiguity $\\text{ker}(\\text{B} \\circ \\Gamma)$ vanishes, then the minima of the loss function are guaranteed to be safe, in the sense that they have return functions that differ up to an additive constant from the true return function. Thus, future work has a theoretical guide for designing practical algorithms, which requires carefully designing the type of partial observability so that the ambiguity is “benign”, and potentially specifying an approximation of the belief matrix B. \n\n> The paper does not discuss other work in RLHF that deals with partial observability and heterogeneity (albeit in a subtly different context), for example [1, 2, 3].\n\nThank you for making us aware of these recent works. We will add the following paragraph to the related work:\n\n“*The literature also discusses other cases of partial observability. [1] and [3] deal with the situation that different human evaluators can have different unobserved preference types. In contrast, we assume a single human evaluator with fixed reward function, which can be motivated by cases where the human choices are guided by a behavior policy, similar to a [constitution](https://www.anthropic.com/news/claudes-constitution) or a [model spec](https://cdn.openai.com/spec/model-spec-2024-05-08.html). [2] assumes that the choices of the human evaluator depend on an unobserved reward-state with its own transition dynamics, similar to an emotional state in a real human. In contrast, we assume the human to be stateless.*”\n\nWe also add paragraphs detailing related work on the modeling choices for human preference selection, truthful AI, and other works that deal with missing information. Please let us know if you would like to see these paragraphs, then we will promptly add them in a subsequent answer.\n\n> Are there easy ways to force the ML model to\"expose the underlying state a bit more\" from time to time? A naive example is forcing verbosity in the case of the deceptive inflation example, but of course one needs to design a more general and sustainable fix.\n\nWe think about this in terms of trade-offs. *In principle*, the environment state could be fully observed by the human: We could just send all the information that the AI receives also to the human evaluators. The reason why we still think our work is necessary is that we expect it to increasingly become prohibitively expensive to show human evaluators all content; after all, human evaluators have limited time, and so if they always fully observe a state sequence, that trades off against the number of labels they can provide within a given timeframe. \n\nThus, the question becomes how to *design* the partial observability in exactly such a way that despite limited information, the feedback-process still leads to correct reward functions being learned. Your idea to “Expose the state from time to time” (e.g. randomly) is interesting, and we are indeed interested in further research that deeply analyzes such settings.\n\nPlease let us know if you have remaining concerns. We will be happy to discuss them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZBriQ4tTpb", "reviewer": "Reviewer_zgLN", "summary": "The paper discusses the challenges that arise when human annotations for RLHF are based on partial observations. They formally define deceptive inflation and overjustification as failure cases caused by partial observation, and theoretically prove that standard RLHF is guaranteed to result in either or both of them. They further analyze how much information the feedback process provides about the return function assuming that human’s partial observability is known and accounted for.", "questions": "Refer to the above comments.", "limitations": "The paper discusses some limitations and future work. In addition to the aspects mentioned by the authors, I am also looking forward to seeing empirical evidences regarding this topic.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The research question proposed in this paper is of significant value. The authors provide a clear formal definition of deceptive inflation and overjustification by introducing policy evaluation function and over-/underestimation error.\n- The paper provides detailed and solid proof over their claims, not only discussing the limitations of standard methods but also showing whether the model can perform better when the human’s partial observability is known. The authors also use many examples and counterexamples to analyze and explain the proposed claims.", "weaknesses": "- The proofs are based on assumptions of a specific MDP structure and a particular human belief function, which might not be easily generalized to realistic, complex environments.\n- No empirical evidence is included in this paper, though real-world examples are given.\n\n(I am not quite familiar with relevant topics and am unable to assess this part.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n> The proofs are based on assumptions of a specific MDP structure and a particular human belief function, which might not be easily generalized to realistic, complex environments.\n\n**It is not true that we assume a specific MDP structure.** Our work applies to any MDP, encompassing all reinforcement learning problems. **Our human belief function is also assumed to be general**: The only restriction is that it sums to 1 over all state-sequences, for a given observation sequence. Future work could think about relaxing the human belief function to allow for \"feature maps\" since humans cannot think about the whole environment state of complex environments. We detail this “feature map” idea [in our answer to reviewer LhS3](https://openreview.net/forum?id=XcbgkjWSJ7&noteId=FWxGuq61x2).\n\n> No empirical evidence is included in this paper, though real-world examples are given.\n\nOur work is indeed theoretical. For our general philosophy, we refer to [our global rebuttal](https://openreview.net/forum?id=XcbgkjWSJ7&noteId=dwv3iH9Ccw).\n\nPlease let us know if you have remaining concerns. We will be happy to address them!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "96cZhAPiHK", "reviewer": "Reviewer_LhS3", "summary": "This work studies the impact of partial observability in RLHF. Two failure cases are defined: deceptive inflation and overjustification, and under them, concrete conditions are provided to impact the learned policy. Moreover, the ambiguity induced by the partial observability is further measured that the feedback determines the correct return function up to an additive constant and a linear subspace, which does not vanishes in many circumstances. Finally, it is recommended that recommended performing exploratory research on RLHF for cases when partial observability is unavoidable.", "questions": "I would love to hear the authors' opinions on the points listed in weaknesses.", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work targets at one important issue in the current RLHF learning pipeline that the human annotators may not observe the full information of the model's generation processes. Taking such scenarios in consideration is vital to keep the trained models trustworthy. I appreciate the authors' efforts in this direction.\n\n- One theoretical formulation is proposed to model how human deals with partial observability (i.e., through the belief matrix). Under this formulation, detailed discussions are provided on what kinds of conditions would lead to impacted policies and how much ambiguity is introduced. The cases of Deceptive Inflation and Overjustification are well aligned with many practical scenarios in my mind. And, the discussed ambiguity is enlightening, also highlighting the need to treat partial observability carefully.\n\n- The overall presentation is very clear, with key messages illustrated both rigorously through theorems and intuitively through examples/figures. I enjoyed reading this work.", "weaknesses": "I am overall satisfied by this work. The followings are a few points that would be interesting if further discussed.\n\n- This work discusses the partial observability mostly in the theoretical sense, with a few hypothetical examples. It would be nice to see or verify the impact of partial observability in real experiments. I understand this would require a large amount of further efforts, while encouraging the authors to contain some discussions (e.g., on experimental designs) to enlighten future works.\n\n- As also mentioned in the limitations listed in Section 7, the formulation of the belief matrix is a bit strong in my mind. I fully understand that it is adopted to deliver the message of treating partial observability carefully, while also encouraging the authors to include some discussions to improve such a formulation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review! \n\n> This work discusses the partial observability mostly in the theoretical sense, with a few hypothetical examples. It would be nice to see or verify the impact of partial observability in real experiments.\n\nOur work is indeed theoretical. For our general philosophy, we refer to [our global rebuttal](https://openreview.net/forum?id=XcbgkjWSJ7&noteId=dwv3iH9Ccw).\n\n## Feature count formulation of the human model \n\n> As also mentioned in the limitations listed in Section 7, the formulation of the belief matrix is a bit strong in my mind.\n\nThis is indeed appears to be a strong assumption. We think further work is needed to evaluate the strengths and shortcomings of this formalization. We want to explain one implicit viewpoint of ours that we think means this formalization might reach quite far. \n Namely, we think it is compatible with how humans intuitively form judgments over events.\n\nConcretely, if you are a human tasked with judging the quality of an observation sequence $\\vec{o}$, what you will do is admittedly **not** to compute an explicit posterior belief $B(\\vec{s} \\mid \\vec{o})$ over $\\vec{s}$ – indeed, if the environment is very complex, it may already be impossible to even think about entire state sequences $\\vec{s}$. More likely, you would think about the presence of certain features in the state sequence; e.g., “how often was there a bug”, or “has the software been installed”, or “how efficient is the code”. One would then implicitly assign a reward to each such feature and then add up the rewards. \n\nThis feature viewpoint could a priori be compatible with our formulation, namely if the feature-based returns implicitly come from a belief over state sequences. This could work as follows: Assume there is a feature map $\\phi(s)$ that maps high-dimensional states $s$ to low-dimensional feature vectors $f$. Assume the reward only depends on feature vectors: $R(s) = R’(\\phi(s))$ for some function $R’$. We obtain:\n\n$$\\sum_{\\vec{s}} B(\\vec{s} \\mid \\vec{o}) G(\\vec{s}) = \\sum_{\\vec{s}} B(\\vec{s} \\mid \\vec{o}) \\sum_{t = 0}^{T} \\gamma^t R(s_t) = \\sum_{f} \\left( \\sum_{\\vec{s}} B(\\vec{s} \\mid \\vec{o}) \\sum_{t = 0}^{T} \\gamma^t \\delta_f(\\phi(s_t)) \\right) R’(f) \\eqqcolon \\sum_{f} N(f \\mid \\vec{o}) R'(f),$$\n\nWhere the outer sum runs over possible features, and where $\\delta_f(\\phi(s_t))$ evaluates to $1$ if and only if $\\phi(s_t) = f$. In the last step, we denoted the big coefficient of $R’(f)$ by $N(f \\mid \\vec{o})$, which is the expected discounted feature-vector count of $f$ upon observing the whole sequence $\\vec{o}$. \n\nThen instead of modeling the human as coming with a reward function $R$ and a belief matrix $B$, we have found an alternative model: we can model the human as coming with a feature-based reward function $R’$ and an expected feature-count $N(f \\mid \\vec{o})$. This highlights that the whole formalism could also be built upon a human model where the human “counts the presence of features”, as long as these feature counts come with consistency properties that allow them to be compatible with a belief over state sequences. We think this viewpoint makes it natural to extend our work to more realistic human models that effortlessly connect with our work.\n\nWe will add this additional discussion to our paper to explain the reach of our belief matrix formulation.\n\nPlease let us know if you have remaining concerns, and we are happy to address them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cd9mM2TD8l", "reviewer": "Reviewer_rT4E", "summary": "The authors present a current problem (research gap): if humans give model answer feedback in a partially observable environment, such feedback may lead to deceptive inflation and overjustification. For example, if humans purely depend on the error reporting result, this will encourage the model to hide the error intentionally, leading to deceptive behavior. \n\nThe authors then introduce the formalization of partial-observable RLHF, and formalize  Deceptive Inflation and Overjustification. For example, turning human beliefs into a matrix B, and replace Return(states) with B$\\cdot$ Return(observation). After that, authors define the concept of ambiguity, a linear subspace. If the distance between human's return function and true return function fall into such subspace, the feedback will be the same.", "questions": "The paper presents a formalization of  Deceptive Inflation and Overjustification and mentions some modifications to current RLHF methods.   Even with proof, the result may not stand without experiments. Thus this paper will grow much stronger if it can be supported from experiments.", "limitations": "The authors admitted that some of the assumptions that this method is based on may not hold, thus need further improvement.\n\nBeyond that, authors may consider creating some metrics for deceptive inflation and overjustification, and then justifying them. In this way, this work could serve as a stepping stone for the community to follow.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "[1]The proposed research question is practical, interesting, and in an early-investigated stage.  The authors formalize this question for human feedback under partial observability.\n\n[2] The authors provide several informative figures, which should be encouraged.", "weaknesses": "[1] No experimental results and metrics\n\nOne main contribution of this paper is the proposed research question and its formalization. Given the formalization, the reader would expect the authors to (1)explain the question within the formalization, (2)give an experiment to empirically reveal the question or justify the proposed solution, and (3)give metrics to judge the effectiveness of methods that aim to solve this question. And the paper only did (1).\nThough this paper gives proof of specific propositions, it's not reasonable for a formalization lacking (2) or (3).\n\n[2] Writing\n\nDespite the clear figures, some logic does not match. The purposes stated ahead of sub-sections are lost,  for example, due to the lack of translation from proven Propositions to the answers to proposed questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review of our work!\n\n> Given the formalization, the reader would expect the authors to (1)explain the question within the formalization, (2)give an experiment to empirically reveal the question or justify the proposed solution, and (3)give metrics to judge the effectiveness of methods that aim to solve this question. And the paper only did (1). Though this paper gives proof of specific propositions, it's not reasonable for a formalization lacking (2) or (3).\n\nOur work is indeed theoretical. For our general philosophy justifying this choice, we refer to [our global rebuttal](https://openreview.net/forum?id=XcbgkjWSJ7&noteId=dwv3iH9Ccw).\n\nWith respect to empirical justifications of proposed solutions, we want to additionally highlight that in Section 6, we sketch the beginnings of a method that is *theoretically justified*. In particular, if $B$ is known, then in Proposition C.5 we prove that there is a differentiable loss function whose minimum contains only feedback-compatible reward functions, which are safe if the ambiguity $\\ker(B \\circ \\Gamma)$ disappears.\n\n> Despite the clear figures, some logic does not match. The purposes stated ahead of sub-sections are lost, for example, due to the lack of translation from proven Propositions to the answers to proposed questions.\n\nThank you for this comment. We will extend the first paragraph in Section 4 to foreshadow Proposition 4.1 in Section 4.1, which, so far, was unmentioned in the intro of Section 4. Our new intro paragraph to Sec. 4 will read:\n\n“*We now analyze failure modes of a naive application of RLHF from partial observations, both theoretically and with examples. In Proposition 4.1, we show that under partial observations, RLHF incentives policies that maximize what we call $J_\\text{obs}$, a policy evaluation function that evaluates how good the state sequences “look to the human”. The resulting policies can show two distinct failure modes that we formally define and call deceptive inflation and overjustification. In Theorem 4.5 we prove that at least one of them is present for $J_\\text{obs}$-maximizing policies. Later, in Sections 5 and 6, we will see that an adaptation of the usual RLHF process might sometimes be able to avoid these problems.*”\n\nFurthermore, we think that the start of Section 5 might be confusing in its current form, where we ask “Assuming the human’s partial observability is known, could one do better?”. This question is only directly addressed in Section 6. To improve the logic, we will make the following changes:\nWe will merge Sections 5 and 6 (i.e., Section 6 becomes the new Section 5.3), and change the second paragraph of the intro to Section 5 as follows:\n\n“*We start in Section 5.1 by analyzing how much information the feedback process provides about the return function when the human’s choice model under partial observations is known precisely. We show that the feedback determines the correct return function up to an additive constant and a linear subspace we call the ambiguity (see Theorem 5.2). If the human had a return function that differed from the true return function by an element in the ambiguity, they would give the exact same feedback — such return functions are thus feedback-compatible. In Section 5.2, we show an example where the ambiguity vanishes, and another where it doesn’t, leading to feedback-compatible return functions that have optimal policies with high regret under the true return function. Finally, in Section 5.3 we explore how one could in theory use Theorem 5.2 as a starting point to design reward learning techniques that work under partial observability.*”\n\nPlease let us know if you see further problems in the clarity of our writing, and we are happy to address them!\n\n> The authors admitted that some of the assumptions that this method is based on may not hold, thus need further improvement.\n\nIt is true that some assumptions on the human model may not hold. Nevertheless, see [our answer to reviewer LhS3](https://openreview.net/forum?id=XcbgkjWSJ7&noteId=FWxGuq61x2): we think a more realistic feature-map formulation of the human model (replacing the belief $B$) would effortlessly connect with our formalization. Thus, we think there is great potential for our formalization to remain relevant even when taking into account more realistic human models. We will add a discussion on this viewpoint in the final version. \n\nAdditionally, it is a priori our belief that the results in Section 4 are robust to more realistic human models, since they are about failure modes that we show to be present even when the human has excellent rational expected-value thinking. \n\nPlease let us know about any remaining concerns, and we are happy to discuss them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Rebuttal

In this global answer, we want to advocate for the inclusion of our purely theoretical work. The reviewers highlight the following (and more!) positive aspects of our work:

- *On the problem setting*: Our setting is “vital” (LhS3) and has “significant value” (zgLN).
- *On deceptive inflation/overjustification*: These are “well aligned with many practical scenarios” (LhS3).
- *On the ambiguity in the return function*: We “Completely characterize the ambiguity” (v92A), which is “enlightening” (LhS3).
- *On the theory*: “very clean” (v92A), with “detailed and solid proofs” (zgLN), and “rigorous [...] theorems” (LhS3).
- *On examples*: Our “many examples and counterexamples” (zgLN) are “very comprehensive” (v92A), and “intuitive” (LhS3).
- *On the presentation*: We include “several informative figures” (rT4E) and have an “overall presentation [that] is very clear” and that they “enjoyed to read” (LhS3).

On the other hand, all four reviewers highlight the lack of experiments and metrics to empirically validate the failure modes of deceptive inflation and overjustification. Reviewers v92A and rT4E furthermore highlight the lack of proposed (practical) solutions or metrics to judge progress toward a solution.[a] We overall agree that these are valuable goals to have, but consider them future work, for our work is meant to lay the conceptual and theoretical groundwork for studying RLHF under partial observability. Our goal was to clearly formalize a problem that has remained unformalized despite being informally known since the inception of RLHF through the [robot-hand](https://openai.com/index/learning-from-human-preferences/), and later the [ELK report by Christiano et al](https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit#heading=h.kkaua0hwmp1d).

There is a large history of published work at NeurIPS, ICML and ICLR that is purely theoretical and comes without practical solutions. We highlight some that are related to our work: In [1], the authors theoretically analyze AI alignment in a setting in which the human’s reward function can change over time. In [2], it is shown that without knowledge of the algorithm with which humans choose their actions, one cannot learn their preferences. In [3], a theoretical umbrella of prior reward-learning work is built. In [4], the authors analyze what happens when the optimized reward function misses attributes important to humans. And in [5], the whole alignment problem is studied in a purely conceptual, pre-theoretic way. All these works are theoretical or conceptual, come without empirical investigations, and without new solutions to the proposed problems. **But crucially, many of these works are well-known and have guided the intuitions of researchers about the problems to tackle in our field.**

Given that similar work has been published before, it is our opinion that our theoretical work should be judged based on the quality of the theory we provide. With the large positive sentiment about all aspects of our work that we quoted above (on the problem setting, deceptive inflation/overjustification, the ambiguity in return functions, theoretical quality, examples, and presentation), we believe that we meet the standard for NeurIPS. We are happy to discuss further in the discussion phase, including in our answers to the individual reviews.

[1] Micah Carroll et al., *AI Alignment with Changing and Influenceable Reward Functions*, ICML, 2024

[2] Stuart Armstrong, Sören Mindermann, *Occam's razor is insufficient to infer the preferences of irrational agents*, NeurIPS, 2018

[3] Hong Jun Jeon et al., *Reward-rational (implicit) choice: A unifying formalism for reward learning*, NeurIPS, 2020

[4] Simon Zhuang, Dylan Hadfield-Menell, *Consequences of Misaligned AI*, NeurIPS, 2020

[5] Richard Ngo et al., *The Alignment Problem from a Deep Learning Perspective*, ICLR, 2024

[a]: While we do not have practical solutions, we **do** have theoretical beginnings of a solution. In particular, if B is known, then in Proposition C.5 we prove that there is a differentiable loss function whose minima contain only feedback-compatible reward functions, which are safe if the ambiguity $\text{ker}(\mathrm{B} \circ \Gamma)$ disappears. This is discussed in Section 6.

Meta-review: This paper raised and formulated an interesting question in RLHF: what if the human feedback is only based only on partial observations. Two failure cases were defined and formalized, corresponding to "deceptive inflation" and "overjustification". It then analyzes how much information the feedback process provides about the return function, and under what cases human’s feedback can determine the return function. I appreciate the study of this new topic, which has been relatively overlooked in RLHF studies, and is important. The overall presentation is also clear. On the other hand, I also agree with some common concerns regarding the lack of experimental evidence of the issue: how serious it is, or does it really occur, in real-world applications of RLHF with real datasets. The paper would have been stronger if it started with such experimental observations, and the current version is a bit more conceptual (which in fact, can be a very good positional paper in my opinion). Given all the considerations, this is really a borderline paper, and the authors may benefit from the reviews this round. I hope the authors can incorporate the feedback in preparing the next version of the paper.
Decision: Accept (poster)