Modeling Human Beliefs about AI Behavior for Scalable Oversight
Authors: Leon Lang
Patrick Forré
Venue: Accepted by TMLR ()

Abstract: As AI systems advance beyond human capabilities, scalable oversight becomes critical: how can we supervise AI that exceeds our abilities? 
A key challenge is that human evaluators may form incorrect beliefs about AI behavior in complex tasks, leading to unreliable feedback and poor value inference. To address this, we propose modeling evaluators' beliefs to interpret their feedback more reliably. We formalize human belief models, analyze their theoretical role in value learning, and characterize when ambiguity remains.  To reduce reliance on precise belief models, we introduce "belief model covering" as a relaxation. This motivates our preliminary proposal to use the internal representations of adapted foundation models to mimic human evaluators' beliefs. These representations could be used to learn correct values from human feedback even when evaluators misunderstand the AI's behavior. Our work suggests that modeling human beliefs can improve value learning and outlines practical research directions for implementing this approach to scalable oversight.

Published in Transactions on Machine Learning Research (08/2025)
Modeling Human Beliefs about Al Behavior for Scalable Oversight
Leon Lang University of Amsterdam
Llang nl
Patrick Forré University of Amsterdam
p.d. d.fforre@uva..t
Reviewed on OpenReview: ht s: o/ vi .nt /f or P g Jf Qn
Abstract
As AI systems advance beyond human capabilities, scalable oversight becomes critical: how can we supervise Al that exceeds our abilities? A key challenge is that human evaluators may form incorrect beliefs about AI behavior in complex tasks, leading to unreliable feedback and poor value inference. To address this, we propose modeling evaluators' beliefs interpret thoir feedback more reliably. We formalize human belief models, analyze their thcoretical role in value learning, and characterize when ambiguily remains. To reduce reliance precise belief models, introduce "belief model covering" as relaxation. This motivates our preliminary proposal to use the internal reprosentations of adapted foundation models to mimic human evaluators' beliefs. These representations could be used to learn correct values from human feedback even when evaluators misunderstand the AT's behavior. Our work suggests that modeling human beliefs can improve value learning and outlines practical research directions for implementing this approach scalable oversight.

---

Published in Transactions on Machine Learning Research (08/2025)
Contents
1
Introduction
3
2 Human belief models
4
2.1 Conventions and preliminaries for linear algebra
5
2.2
Preliminarics on Markov Decision Processes
2.3
The human's ontology and reward object
2.4
The human's feature belief and observation return function
2.5 Connection to the problem of scalable oversight
8
2.6
The definition of human belief models
2.7
Complete belief models and the ambiguity
2.8
Faithful belief models
2.9
Conceptual examples
15
3
Human belief model covering
17
3.1
Human belief model covering and its implications
................................
18
3.2 Morphisms of human belief models and ontology translations
19
3.3 An example of symmetry-invariant features and reward functions
21
3.4 A proposal for belief model covering practice
Discussion
29
4.1
Summary
................................
29
4.2 Related work
................................
29
4.3 Future work
31
4.4 Conclusion
32
References
33
A Preliminary results on linear algebra
41
B Balanced human belief models and choices
44
C A diagram in the category of human belief models
46
D Details on the example with invariant features
51
E Mathematical interprotations of related work in our framework
54
2

---

Published in Transactions on Machine Learning Research (08/2025)
1 Introduction
In recent years, reinforcement learning from human feedback (RLHF) (Christiano et al., 2017) and varia- tions like direct preference optimization (Rafailov et al., 2023) have become practical approach for aligning language models (Ziegler el al., 2020; Stiennon el al., 2022; Bai et al., 2022; Ouyang el al., 2022). These tech- niques have then been used in the aligmment of large-scale foundation models like ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI et al., 2024), Gemini (Gemini Team, 2023), Llama (Touvron ct al., 2023; Grattafiori et al., 2024), and Claude (Bai et al., 2022; Authropic, 2023a;b).
RLHF uses feedback of human evaluators on Al behavior to give information about desired behavior. But as AI systems become more capable, they may eventually outstrip our ability to evaluate them. The problem of scalable oversight therefore asks how to effectively teach our preferences to AI systems when they become more capable than ourselves (Amodei et al., 2016; Christiano et al., 2018; Trving et al., 2018; Leike et al., 2018; Bowman et al., 2022). Conceptually, if human evaluators lack the capacity to fully understand the AI's behavior, then this may incentivize the AI to perform behavior that the human evaluator only believes to be good, possibly at. the expense of actual value.
Recent work has in fact revealed problems with RLHF that stem from erroneous human beliefs about Al behavior. In early example (Amodei et al., 2017), an AI was supposed to grasp ball with robot hand in simulation. The evaluators, who looked at video of the behavior, sometimes believed that the hand was grasping the ball when it was in fact only in fromt of it, leading to positive feodback for the wrong behavior. In Cloud et al. (2024), a synthetic overseer in a baseline setting is unaware of whether a reached goal is a diamond (positive) or ghost (negative), leading the policy to learn to approach the ghost. Denison et al. (2024) shows a language model modifying a file unbeknownst. to a synthetic evaluator, successfully docciving it into believing a checklist has been accomplished. Similar models have also been shown to mislead humans who are limited in their time competence to evaluate question answering or programming tasks, leading them to believe that the performance is better than it is (Wen et al., 2024).
Given the crucial role of evaluator beliefs in erroneous feedback, in this theoretical work, we are exploring the idea to model human beliefs about AT behavior. Our idea is that if we knew what the human believes the Al has done in its environment, then we could properly relate the human's feedback to that believed behavior instead of to the actual behavior. For example, imagine the robot hand from Amodei et al. (2017) is in front of the ball and the human evaluator gives positive feedback. Additionally, assume we know the human believes the ball was actually grasped. If so, then we know that the human thinks grasping the ball is positive, and can incentivize this behavior from now on.
But what are beliefs? In classical work on partial observability, beliefs are formalized as probability distri- butions over environment states (Kaclbling et al., 1998), possibly including models of other agents (Gmy- trasiewicz& & Doshi, 2005). In Lang et al. (2024), human evaluator beliefs are given by probability distribu- tions over state sequences in the AT's environment. They show theoretical toy examples that modeling such beliefs can help to learn effectively from feedback. However, in reality it is unrealistic that humans form probability distributions over entire state sequences, and it would be prohibitive to specify such belief explicitly. We think it is more realistic that humans think in terms of features, which we concept as higher-level and independent properties of trajectories in an environment. We then model a human belief about an Al's behavior as a vector of feature strengths.
This leads to our notion of a human belief model, which we introduce in Section 2. This model, together with feodback on obscrvations, then thoorctically allows infer the return function over trajectorics up to an inherent ambiguity, which we characterize in terms of the human belief model. The ambiguity disappears when the model is complete, which holds, in particular, when the human's beliefs of all observations linearly cover the feature combinations that are possible in the environment (Theorem 2.13). We then analyze conceptual toy examples of non-complete and complete models and consequences for the resulting return function inference.
We would like to use human belief modeling in practice to make concrete progress on scalable oversight. However, realistically, we cannot model human beliefs precisely; after all, we do not even have an explicit specification of the human's feature set! Tn Section 3, we thus relax the requirement of exact modeling
3

---

Published in Transactions on Machine Learning Research (08/2025)
by investigating belief models that can represent all return functions and feedback functions that are compatible with the true belief model. We then say that these models cover the true belief model. When such a model is complete, it can replace the true model for the return function inference (Theorem 3.2). In a conceptual example, we analyze a human with symmetry-invariant features, which we can cover with a complete model that assumes symmetry-invarient reward functions.
We also characterize model coverage by the existence of what we call model morphisms and linear belief- compatible ontology translations (Theorem 3.5). Intuitively, this means that if one can linearly translate from the specified model's features to the human's features in way that is compatible with their beliefs about observations, then coverage holds. The notion of a linear ontology translation is strikingly similar to work in the field of mechanistic interpretability on sparse autoencoders (Cunningham et al., 2023; Bricken et al., 2023), which linearly map from the internal representation space of language models to a space of human-interpretable features.
Inspired by this connection, in Section 3.4, we make a preliminary proposal to use adapted foundation models to construct belief model that covers the true human belief model. The resulting training scheme would work like typical reinforcement learning from human feedback, but use adapted foundation model with internal representations that mimic the human's 's beliefs of the AI's behavior. After learning the reward model, it can then be applied during policy optimization by using a more capable foundation model for creating the internal representations. Our theory specifies precise conditions under which the procedure leads to a correct return function. This training scheme does not assume the human to fully understand the behavior they evaluate and is not based on amplification of the human's capabilities, which we think provides new angle to make progress on the problem of scalable oversight.
Finally, in our discussion in Section 4, we summarize our work, survey related work, motivate theoretical and empirical future work, and conclude.
2 Human belief models
In this section we define the notion of human belief models that can help to infer return functions from the human's 's feedback.
In Section 2.1 we start by explaining our conventions and some preliminaries for linear algebra. We recom- mend also experienced readers to briefly skim this scction to understand our notation. In Section 2.2, we briefly define Markov decision processes and slightly adapt them: We do not make the assumption that the return function over trajectories comes from a reward function. This allows us be slightly more general. In Section 2.3, we then introduce the notion of the human's 's ontology, which is map that assigns vector of feature strengths to each trajectory. Our crucial assumption is that the return function evaluates trajectory by summing up the rewards of each feature, weighted by the feature strengths. This allows us to recover classical reward functions as special case.
In reinforeement learning, the goal is to maximize the return function, but we assume this function to be unknown: It represents the implicit values of a human evaluator and needs be inferred from the human's feedback: In Section 2.4, we explain the human to give feedback based on forming (possibly incorrect) belief over features from observations. We call the resulting feedback the observation return function. In Section 2.5 then explain how our problem setup connects to the general problem of scalable oversight and motivate our solution approach: It is based on leveraging human belief models to better interpret the human's 's feedback; we introduce them in Section 2.6. When a model of the human's beliefs is known, one can infer the return function from the human's feedback up to an ambiguity, which we characterize in Section 2.7. We also define complete human belief models, for which the ambiguity vanishos, and characterize them in Thcorem 13. In Section 2.8, we introduce the dual notion of faithfulness. In Section 2.9, we study conceptual examples that give an intuition for when the ambiguity does, does not, vanish.
4

---

Published in Transactions on Machine Learning Research (08/2025)
2.1 Conventions and preliminaries for linear algebra
Let X be set. For X, we define the delta function 82 X R usual by
1, x'=x,
(x(2') 0, clsc
Let R be the real numbers, Then R* denotes the vector space of functions from X to R, which one can also view as column-vectons indexed by z E X. For a function v E R we write v(x) for the entry at position a X. The standard basis functions of Rx are given by fex) with 6x = ôg. Then every function v E R can be written as =mex v(x) Ex- We define the standard scalar product (,,) R x RX R by
(v,w)= = Σ v(r)w(())
ZEX
Let Rx + R be a linear map. Then we view also as a matrix with matrix elements Ayr == [A(e))] (v) R for I yeY. We write Ay E R for the row of A at index y Y, which is the function with entries (z( Aye- Consequently, if E R and y E EY, then obtain
[A( E (())(A(e))(()) E Agxv(z) (Ag, v). ZEX
zeX
This corresponds to the typical way that linear functions can be represented as matrix-vector products. If V S Rx is a vector subspace, then we write Alp : + R for the restriction A to V, which simply given by (Alv)(v) = A(v) for all u NEW V. We denote the kernel and image of A R - RY by
(A) = UUER
A(v) = CR., =
im{ = SWWRR B - =w
Le., they are the set of functions in R that are sent to zero by A, and the set of functions R that are hit by A, respectively. Both are vector subspaces of their respective surrounding vector spaces. We explain basic properties of kernels and images in Appendix A.2 and will refer to those results when needed.
Sometimes, our linear functions "come from" a deterministic function in the other direction. Le., if Y + X is a function, then we define the linear dual function h² R + R for E R and E Y by
[b'())] v (h(y))
(1)
The matrix elements are then given by
1, h(y) = x,
11 = [ea())]](v) = en (n(v))
(2)
0, else.
For two linear maps 4: A:R. RY and B RY R write their composition B A R which has matrix elements (B A).z Eyer By for E X, 2 Z. This also implics
R
(Bo A)= EB B
(3)
DEY
Finally, whenever we draw diagram of (usually lincar) functions in this paper, the diagram commutes, meaning that all directed pathways from one node another node are the same function. For example, diagram of the form
5

---

Published in Transactions on Machine Learning Research (08/2025)
R
A
R
R
we have C= B o A. The only exceptions to this convention are Equations (22) and (28) in the appendix.
2.2 Preliminaries on Markov Decision Processes
Much prior work involving human feedback, especially in RLHF, considers the context ual bandit frame- work (Ziegler et al., 2020). However, we are motivated to contribute to a theory that will apply to advanced AI systems interacting with more complex environments, as this is where we expect most of the risks of advanced AT to be. We thus work in the setting of an MDP (S, Po. T, G),,, with finite set of states S and actions A, transition kernel T S A A(S), an initial state distribution Po ASS), a finite time horizon I' E {0, 2,...... and the human's 's implicit return function G, which we clarify after defining trajectories below. We fix this generic MDP for the rest of the paper.
We now define trajectories. In the rest of the paper, whenever we have state-action sequence & E (S x A) x S present in some context and then write about states and actions 80, - -1, -,, 87, then they implicitly refer to the states and actions in g. Then the set of trajectories be given by
E= {s (8 x A) S & is possible } S (8 A) x XS,
where we call state-action sequence possible if Pa(so) > 0 and T(s: 81-1, > 0 for all 12 1. Then the return function is a function G E IR.
Note that this formalism does not assume that decomposes into a reward function R: R Ax R. other words, the formalism allows for human preferences that do not satisfy the reward hypothesis (Bowling al., 2022) and is thus slightly more general than typical reinforcement learning settings.
Policies are functions π S A(A). A policy π together with the MDP induces distribution P A(E) over trajectories by sampling initial states from Po, actions from n, and transitions from T. The policy evaluation function is then given by
J(T) = E [G()]]
(4)
(-P)())
The goal in reinforcement learning to find a policy that maximizes this evaluation function.
2.3 The human's ontology and reward object
We assume that the return function E R encodes what a given human cares about. We imagine that G measures the quality of cach trajectory linearly in certain feature strengths associated to each trajectory; here, features are high-level return-relevant concepts.
More precisely, we assume the human comes equipped with a finite feature set Ω. The human's ontology is a function E + R that encodes for each trajectory & E E and feature w E Ω the extent [X(E]] (w) to which the feature w is present the trajectory ç. As we discuss greater detail in Section 2.4, these feature strengths X(E) are idealized, i.e., the human may not be able compute them. In the general theory, we allow them to be negative, though it may help imagine them to be non-negative. The human's reward object is fixed function Rq R that assigns to each feature E Ω the degree Re(w) to which the human
likes this feature. The return function G : > R evaluates a trajectory by summing up the quality of all features, weighted by the extent to which they appear in the trajectory:
G( = Σ [W) Ro(w) (A(E), Ra).
(5)
WEn
6

---

Published in Transactions on Machine Learning Research (08/2025)
Let A : R - R be given by (E) (A(E), Ra). Then A is linear function from which we can recover À using the matrix elements of A: [>(E)] (w) Aew (Proposition A.1). We slightly abuse the terminology by referring to both X = + R and A R +R as the human's ontology. This representation of the ontology satislies (Ra) = G.
Example 2.1 (Classical reward functions). Assume that the human's feature set is Ω = S x A x S, i.e., the set of all state-action-slate transitions. Then the reward object Ra = R E RSxAxS is a reward function in the classical sense. Let 7 E [0, 1] be discount factor and define the ontology A = r RSXAXS 7 R by
[T)R)]
Σ y 0c, t=
G ((R) -RR is then given by the discounted sum of rewards of individual transitions in trajectory, as is typical in reinforcement learning. The malriz elements of the ontology I are given by
Tg.(,,,,'') [((
Σ r (s,,
au 8641)
Σ 7 (',,,'''') ae, 8t-1),
In other words, the extent to which the "feature" (s, a, s') is present in the trajectory & is simply the discounted number of times that it oppears.
Remark 2.2 (Lincarity assumption). Equation (5) encodes that the true return function is linear in the features of the human's ontology. This is an important assumption that simplifies our theory substantially, but it also means that our resulls might hinge on the assumplion, being true. Is this justified?
First of all, linearity assumptions in reward learning common, for example in Ng et al. (2000); Abbeel & Ng (2004); Ziebert et al. (2008) for inverse reinforcement learning or in parts of Song et al. (2024) for the case of preference fineluning. Additionally, a priori, our linearity assumplion is quile flexible. For example, if the feature set is large enough, then any function can be considered linear in such features, as the case of classical reward functions in Example 2.1 1 demonstrates: In that case, the return functions are linear in features given by state-cction-stat transitions.
Yet later in this work, don't only want to theoretically assume the human's return function to obey linearily assumption we also want to be able to concrelely model the underbying features. If the features that allow for a linear return function get increasingly complex, we expect it to become more difficult to model those correctly in practice. We thus consider it an open question to what extent assuming linearity is appropriate, and we hope for fubure theoretical work that relazes this assumption (ef. Section 4.3).
2.4 The human's feature belief and observation return function
In standard reinforcement learning, it is typically assumed that we know the return function G to be max- imized by a policy. However, is the implicit return function of a given human. We must thus rely on feedback by the human to infer information about G. A naive idea would be show the human each tra- jectory ç, ask them to evaluate it by computing G(E), and to use these returns to train a policy. There are three challenges with this plan:
(i) The human might not have access to the full trajectory & or all trajectorics & E E. For example, in complex environments, they would usually only receive partial observations.
(ii) Even if & were fully accessible, the human might not have the capacity to compute the features X(E). For example, if & encodes a proof-altempt of a mathematical conjecture and w encodes "correctness" then the human may not have the competence to determine the extent [X(E) (w) to which & is correct.
(iii) Even if the human could fully compute X(E), they may not have full access to their reward object Ra in order to then compute G(E) as an explicit number.
To address (i); we assume set of observations O that the human can receive. They may come from an observation function O: = > O, as in some examples from Amodei et al. (2017); Lang et al. (2024); Cloud
7

---

Published in Transactions on Machine Learning Research (08/2025)
et al. (2024); Denison et al. (2024). For example, O(E) could be sequence of observation segments, one for each time-step. Alternatively, O could be set of simulations to probe the human's 's opinion on specific situations that may be present in real trajectories. We can also imagine O to be a subset of E. For example, it can make sense to only show trajectories to the human if they are able to correctly evaluate them, as in easy-lo-hard generalization ion (Sun el al., 2024; Hase el al., 2024; Ding el al., 2024). In any case, we assumee O to be a fixed set of observations where, for the most part, we are agnostic about the process that generates them.
To address (ii), we assumee the human then has feature belief function, i.e. function E: O R that encodes for each o E O the extent (w) to which the human believes the feature w to be present in the observation o or an associated unknown trajectory &. Even if O = =, we can have o X, which reflects that the human may believe the features of a trajectory to be different from what they actually are.
The human then judges observations o E O according to the observation return function Go E R It evaluates an observation by summing up the quality of all features, weighted by the exlent to which the human believes them to be present:
Golo)= = Σ [c])]] Ra(w) = (c(o))Rn))
(6)
WEn
Let E: R + RD be given by (c) = (e(o), Ra). Then as for A, E is a linear function from which we can recover using the matrix elements of E: [<(o]] (w) = En (Proposition A.1). We again slightly abuse the terminology by referring to both e: O - RD and E RR + R as the human's feature belief function. This representation of the featre belief function satisfies E(Ra) = Go.
To address (iii), in RLHF it is typically assumed that the human can make choices between observa- tions (Christiano et al., 2017) that implicitly reflect the undorlying return function (or, on our case, ob- servation return function). We discuss this setting in Appendix B for the special case that all relevant. linear functions are row-constant. However, in the main paper, we do not want to overcomplicate the theory and assume that the human can directly compute Go. In principle, if we would show cach E O to the human, we could then record the entire observation return function Go. Thus, we assume Go, which represents the human's 's feedback, to be fully known.
Remark 2.3 (Comparison to other work involving partial observations). Since we work with a set of ob- servations O, we should clarify that we do not work in a classical POMDP setting. In POMDPs (Kaelbling et al., 1998), the agent only partially observes the environment, and may then keep track of belief state for making decisions in the environment. and macimizing reward. In contrast, we work in an MDP selling in which the agent's 's policy x S + ((A) fully observes the environment state S, see Section 2.2. It is the human evaluator who may obserce the environment only partially for evaluating the policy.
More closely related is the framework of I-POMDPs (Gmytrasiewicc & Doshi, 2005), which is multi-agent setting in which each agent maintains a belief over not only the environment, but also over other agents and their beliefs. Recent work on partially observable assistance games (Emmons el al., 2024) provides a special case that is more similar to our setting by considering an agent and a human in a shared environment in which the agent is trying to marimize the human's latent reward function. In that work, the hummn is assumed to form a calibrated belief over entire environment states, whereas we consider potentially faulty beliefs over trajectory features. We think it might be useful to allempl synthesis of both sellings (ef. Sections 4.2
and 4.3.1).
2.5 Connection to the problem of scalable oversight
In this interlude, we explain how the sctup that we developed over the previous subsections relates to the problem of scalable oversight (Amodei et al., 2016; Christiano et al., 2018; Irving et al., 2018; Leike et al., 2018; Bowman et al., 2022). We also motivate our solution approach of modeling human beliefs.
Scalable oversight is the general problem of how to oversee AI systems when they are more capable than the human overseers. While it is difficult to directly model the capabilities of AI systems, we can at least note that highly capable AT systems will plausibly act in large-soale environments and perform behaviors whose
8

---

Published in Transactions on Machine Learning Research (08/2025)
precise meaning may not be adequately understood by humans. This can be modeled in the setting that we developed earlier:
Scale: If the Al acts in an environment with trajectories & E E, hurnan overseers may not be able to observe the entirety of the trajectories. We model this by a set of potentially smaller observations o E O on which humans give feedback.
Understanding: We model the human's understanding of an observation by its feature belief e(o) E R In cases where an observation o E O belongs to a trajectory & E E in the environment (e.g., by observing a subset or other transformation of the trajectory), gap in the human's 's understanding can be characterized by e(o) + A(E): The human's understanding of the observation is different from the true features of the corresponding trajectory.
How do these assumptions impact the return function inference? Naively, if we did not properly account for the human's limitations, we might assume that the human directly observes the full trajectory & and assesses it correctly. If we then try to learn a return function G E + R and obscrve the human's foedback Go(o), we make the inference G(E) = (o) = (e(o), Ra), which can differ from the true return G(E) = (X(E), Ra) since (()) * X(E). Now, assume there a feature w Q that very bad (Ry(w) < 0) and present in the true trajectory ([A()) (w) > 0), but not in the human's belief of the corresponding observation ([e(o) (w) = 0).
Then this feature leads to large negative contribution to G(E), but not to Go(o), which can lead to imationimation of the inferred return for g:
G(E) = (elo). > (X(E), = G(E).
Thus, in the policy optimization phase using the inferred return function G, & may be positively reinforced, leading to a policy with potentially bad performance as judged by the true return function G. This phe- nomenon is analyzed in a special case under the name deceptive inflation in Lang et al. (2024), and might also lie at the heart of the empirical failures observed in Amodei et al. (2017); Denison et al. (2024); Wen et al. (2024); Williams et al. (2024).
In our work, we altempt to solve this problem by viewing Go(o) = (e(o): Ra) not as an estimate for the true return of a trajectory &, but instead as evidence for the human's reward object Rn- The idea is that we assume to know the human's faulty belief c(o), which means that Go(o)
(c(o), Ra) is lincar equation that can be used to narrow down the space of possibilities for Ro. Once Ro is sufficiently determined, and if X(E) is also known, we can then infer the true return function G using the equation G(E) = (A(E), R2). This approach complements other work on scalable oversight, which usually falls into the regime of amplified oversight (Leike et al., 2018; Saunders et al., 2022; Irving et al., 2018). Such work attempts to improve the human's ability to give accurate feedback, whereas our work atlempts to learn the correct return function from potentially inaccurate feedback by leveraging additional information in the form of a known model of the human's beliefs. We then get closer to reality by relaxing the assumption of knowing the human's 's beliefs in Section Section 3.
2.6 The definition of human belief models
In order to infer the return function G from the human's feedback Go, we assume the features Ω ontology A, and feature belief function & are all known; additionally, we assumee may have priori knowledge of vector subspace of valid reward objects V S R in which Ra resides: Ra E V. This may come from any a priori knowledge, e.g. of certain symmetries in the environment that leave rewards invariant (cf. Example 2.19 and Section 3.3). This leads to the following notion:
Definition 2.4 (Human belief model, representing). Let be the set of trajectories in an MDP and the fixed set of observations that human evaluator receives. Then a haman belief model (or belief model, or model, for short) is a tuple M = (2, V), where:
Q is set called feature set;
AA: R R is linear function, called ontology;

---

Published in Transactions on Machine Learning Research (08/2025)
E: E:R + R is linear function, called feature belief function:
and V C R is vector subspace, called space of valid reward objects.
A human belief model M represents the true return function G E R and observation return function Go O R if there is reward object Rn E V with G = (RR)) and Go = ERRa).
When appropriate, with [c(o]] (w) Eas of the ontology and feature belief function, respectively (ef. Proposition A.1). We visualize model M = (92, A, E,
will also use the representation A = R with (w) Aew and e: O - R
R
R =
R Ul
R
V
RO
R
where we the latter version in the special case that V RR.
Remark 2.5 (Which human belief model?). A priori, there can be many human belief models M
(Ωt, K, E, D) that can represent the true reburn function and the human's feedback Go. For ezample, in Sec-
tion 3.3 we will discuss a situation with three different models, and in Appendiz C one with many more. In
fact, there is always a trivial belief model that can represent G and Go: View R as the R-vectorspace R of functions from one-element set {*} to R. Associate to G + R = R(+ and Go O + R R(*
via Proposition A.1 the linear functions lin (+) + RR² and lin(Go) R (+) + RR given by
[lin(G)) (r- ex) = r G, [lin(Go)] (r-eer)))))))) e.) r. Go
for R. and the only basis vector e.. Then, define the human belief model
M ({+), Gin lin(Go), R(+))
with the feature set {+}. This represents and Go with the reward object e, E R(+) since [inn = and [en(GG)](e.)) = Go Intuitively, the feature then means "goodness", the "ontology" " G = R (*) measures the eztent to which "goodness" is present in trajectory, and the "feature belief function" Go O + R(+} measures how much goodness the human believes to be present in an observation. This trivial model not very interesting for our purposes since assuming that the human belief model is known would then presuppose knowledge of G itself which the return function that we want infer based on feedback Go-
A belief model M = (2, V) is more interesting if Ω consists of variety of meaningful concepts such that G and Go decompose linearly over these fealures, in such way that:
(i) It is casier to know the model M than to know the return function G;
(ii Given M and Go, it is easy to determine G.
For the rest of this section, imagine to know the "true" belief model M, which assume to have these properties. Note that these are philosophical assumptions that guide how we talk about M and how we imagine its application. But mathematically, all of our claims here and in the upcoming sections are correct whenever M represents G and Go-
In Section 3, we then relax the requirement (i) of "knowledge" " of M, and specifically Section 3.4 we make a proposal for model specification using foundation models. We then also show how G could in principle be learned by learning Go via supervised learning.
10

---

Published in Transactions on Machine Learning Research (08/2025)
Example 2.6. Assume the setting from Ezample 2.1, in which Q = A and A = P R - R In this ezample we add fealure belief function to specify full human belief model.
Assume the human comes equipped with a belief function b: O + RR that assigns a probability distribution over trajectories b(o) E A(E) R to each observation o E O. For ezample, this could be a posterior belief if o is sampled by first sampling trajectory according some policy and then computing the observation o O(E) for on obervation function O = + O. Then we assume that the feature belief function O RXXAXS computes, for each O and s') S A S, the expected discounted number of times that the transition (s, a') appears in the trajectory & that gave rise to observation
[s(o)](s,a,s))) = Σ (S) Te.(s.a,))
SEE
We now determine the matric form of c. Let B R² + RO be the linear function given by [B(G)] = (b(o), G), which has matriz elements Bog [b) by Proposition A.1. We obtain
Ea((aa = (s,,,s)
Σ EE
Bog Ter(S.a,))) (BoT) (BoT). (s.a.'))
Thus, 3 BoT.
Finally, assume that the set of valid reward objects is simply given by V RSXAxS Then, in total, our model is given by (9, A, E, V) = (8 x Ax r, BoT, RSXAXS):
R
RSAxx
Bor
R
One can compore this with Long et al. (2024) and realize that our result coincides with their belief model, thus showing that generalize their work.
2.7 Complete belief models and the ambiguity
In this whole subscction, we fix an MDP with trajectorics E, obscrvations O, and corresponding return function G E I and observation return function Go R We also fix human belief model M (2, A, E, V) that represents G and Go with an implicit reward object Ro V: A(Ra) G and E(Rg) Go.
We can visualize the model together with the reward object and return functions as follows:
G
R
R Ul
Ra
E
V
Go
Ro.
We are concerned with the following question:
Question 2.7. Assume the human belief model M and the human feedback, the form of the observation return function Go, are known. Under what conditions, or what extent, can we infer the return function G
11

---

Published in Transactions on Machine Learning Research (08/2025)
In Section 3, will relax the assumption that the belief model is known precisely.
The idea for the answer is as follows: When trying to infer G from Go. we make use of the knowledge that they are connected by the unknown reward object Rg E V. Thus, one can first try to determine & reward object Rq E V/ with the correct observation return function (Ra)) = Go. Then the question is whether the corresponding return function A (Ra) equals the true return function G. They differ by the return function G' = A G. The ambiguity will then be defined as the set of all these differences, and the question will be how to characterize it and when it vanishes, leading to the notion of complete human belief model.
Desinition 2.8 (Teedlack-compatiinn ambiguity). We define the set of relurn funclions that are feedbbck- compatible with Go
(GO) = G aR E V: E(Ra) Go and A(Ra) - =].
We define the ambiguity left in the return function G after the observation return function Go is known by
(G, = GR G²= for E &
Then clearly, we have
(Go) Amb
This leaves open the question of how to characterize the ambiguity and when it vanishes. Proposition 2.9 (Ambiguity characterization). We have
(G) AmbM(G, Go) = A(ker(E) nV)..
Proof. For one direction, let G' E Amb Ammb Go). Then G' = A((R) - for Ro E V with E(Rn) = Go. By the linearity & we obtain
E(RA Ra) E(Ra) Go =0
and thus Rn- Ra E <ke(E)) n.. Since A is linear, obtain
G' A( - G A(Ra) A(Ra) A(Ra Ra) A(ker(E) n V).
For the other direction, let G' E A(ker(E) n V). Then G' = A(R))) for Ry ker(E) nv. Define Ro Rn + R² E V. By the lincarity of & and by R² ker(E) we obtain
E(Rn) = E(R)) + E(R)) = Go + E(R)) = Go.
Finally, the linearity of A shows
G = (A((R))= A( Ra) = A()a) - A(Ra) = A(Ra) - -G.
This shows G' E Mub
See Proposition B.6 for a version of the preceding proposition where the feedback is given by a choice probability function instead of Go. See Appendix C.4 for the ambignities of many human belief models.
Remark 2.10. In light of the previous proposition, it turns out that the ambiguity does not depend on the true return function and observation return function, and we can thus simply write it as Amb
Example 2.11. We continue Example 2.6, with the model given by M -= -S A x S,T,B. Be r, RSAAX)). Proposétion 2.9 implies that the ambiguity is given by Amb = [ = ker (B) n (T)) Interested readers can find characterizations of this ambiguity in examples and special cases in Lang al. (2024). See Sections 2.9 and 3.3 for self-contoined ezomples of ambiguity characterizations in our work.
It important to know when the ambiguity disappears, which is captured completeness the following definition.
12

---

Published in Transactions on Machine Learning Research (08/2025)
Definition 2.12 (Completeness). We call the human belief model M (2,, complete if ierr(E)) S ker(A) Le. if Amb (ker(E) n V) 0 (ef. Proposition 2.9).
We will explain in Interpretation 2.14 why call this property completeness. We find equivalent and sulficient conditions of completeness in the following theorem:
Theorem 2.13 (Completeness characterization). Remember the representations : R and e O RR of the ontology A TR R and feature belief function & TR R respectively- Consider the following four statements:
M is complete.
There ezists a linear function R + + with A/y = Z E
There exists a linear function Z R R with A Z E.
4. For all E E, X(E) E R is contained in the spon of the image of e: X(E) { Eveo Zoc(o) Zo RJ.
Then 1 and 2 are equivalent, and 3 and 4 are equivalent and imply 1 and 2:
3 4
Furthermore, the linear function Z from 2 and 3 satisfies G Z(Go). Finally, if R then all four statements are equivalent.
Proof. Note that ker( (er(A)n) = ker(A/)) and ker(E) n V= ker(Elv). Thus, the equivalence of statements 1 and 2 immediately follow from Proposition A.3. That 3 implics 2 is obvious.
Now assume statement For all & E and o E O, let Z E R. be coefficients with
X(E) = Σ (o)
oEO
This implies
Agw Σ oEO
ZgoLos.
Define the linear function RD + R to have the matrix elements Zzo. Then the previous equation
implies A 7 E, proving statement 3. That implies 4 follows by reversing these arguments.
Assume statement 2. Then
= Alva)(R)) = (Z o Elv) = ((E(Ro)) Z(Go).
That all statements are equivalent if V = R is clear.
Interestingly, for complete models, the preceding proposition shows that G Z(Go) for linear function Z that only depends on the human belief model M, thus showing once more that we can infer G from Go if the model is complete. Since Z may be impractical to find, it may however be more useful to determine G as the unique deedback-compatible return function by first finding Ra with E(Re)= Go. Go-
Interpretation 2.14. Overall, we have thus answered Question 7: When the hamnan belief model is known, then can inferred from Go up to the ambiguity Amb = A ((kerEE) n V), which vanishes if the model M is complete. Looking back at the definition of feedback-compattbe return functions, G then determined G A ((Ra) for any Ra V that gives rise to the humon's feedback: E E(RR) = Go. For completeness, we then found further equivalent and sufficient conditions in Theorem 2. 13.
13

---

Published in Transactions on Machine Learning Research (08/2025)
We now explain why chose the name completeness. By Theorem 2.13, stotement 4, sufficient condition for completeness is the eristence of coefficients Zeo E IR such that for all E E, we can urile X()) as linear combination of the e(o):
X(5) Σ ZzoE(O).
This means that the dnservation-dependdee feature beliefs c(o) span the true feature strengths of trajectories. In this sense, the belief model is "complete": The human's interpretations of observations sufficienlly entail true features of trajectories in the MDP. It is thus not a surprise that completeness implies that the ambiguity disappears, and thus that we can infer the return function from the observation return function Go.
2.8 Faithful belief models
Let again M = (St, A, E, V) be human belief model for an MDP together with trajectorios E and observa- tions O, and let Ra E V with G = (RR),, Go = (Ra). Here, we briefly study the notion of faithfulness of human belief models, which is dual to completeness. cannot be distinguished by their return functions should also not differ their observation return functions if the human's feedback reflects a belief over the actual trajectories in the MDP. We now define the notion and then come back to this interpretation in Interprelation 2.17.
captures the idea that any two reward objects that
Definition 2.15 (Faithfulness). We call the human belief model M faithful if (er(A)) nV C ker(E)
Many human belief models study in this paper are faithful. The model from Example 2.11 is faithful since kTr(T) S ( All models from Scction 3.3 faithful. In Appendix C.2 we study several more faithful human belief models. We can find a characterization of faithfulness that is completely dual to Theorem 2.13:
Proposition 2.16 (Faithfulness characterization) Remember the representations A: R and e: O + IR of the ontology A : IR + R and feature belief function & : IRa - R respectively. Consider the following four statements:
1. MM is faithful.
2. There erists linear function Y R R with E1y = Y Alv
There erists linear function R > R with E
4- For all E O, c(0) E R is contained in the spon of the image of X: c(o) E { EEEE (GN(E) Ygg E RR.
Then 1 and 2 are equivalent, and and are equivalent and imply and 2:
3
Furthermore, the linear function Y from 2 and 3 satisfies Go Y(G). Finally, V = R then all four stalements are equivalent.
Proof. The proof is analogous to the one for Theorem 2.13.
Interpretation 2.17. By Proposition 2.16, sufficient condition for faithfulness is the existence of coeffi- cient Yog E IR such that for all E O, we can wrile e(o) as linear combination of the X(E):
E(O) Σ YogA(E). EEE
We can suggestively interpret Yog as the human's "belief" that the true trajectory & was responsible. for the observation o.! Under this interpretation, the feature strengths c(o) are like an "expected value" of true
Thouuh that Y is usually not unique and E R² usually actual probability distribution over g.
14

---

Published in Transactions on Machine Learning Research (08/2025)
feature strengths, given the human's beliefs. Thus, the feature beliefs encoded by "faithful" to belief over the actual MDP. This interprelation is especially wolid for Ezample 2.6, where Y is given by B, and thus Yog = Bog = (E) is the probability of the trajectory &, given o.
Faithfulness is also philosophically reasonable for a related reason. Assume RR, Ra E are two reward objects with A (Ra)= = (Ra)) Then the resulting return functions coincide in their evoluation of all trajectories. It would then be reasonable to assume that they also coincide in their evaluation of obsercalions, insofar as observations give information about a state of affairs in the actual MDP: (Rg) = E {(R)). This requires that A ( Ra) O implies (RR Ra) = 0, which eractly means that ker(A) n S ker(E) O n.,, i.e. the faithfulness of the model. However, since human evaluators do not necessarily have feature beliefs that ralional to this exlent, and since this properly is not needed in the rest of our theory, we do not assume faithfulness of our human belief models in the generol theory.
2.9 Conceptual examples
We now present some simple conceptual examples to illustrate the theory, and espocially the ambiguity. Wo will not define entire MDPs but only those parts the formalism that are necessary to reason about the ambiguity. In all our examples, we assume O C E, i.c., observations are entire trajectories: They are fully observed. This brings other factors than observability into the focus, like the human's understanding of the trajectories, and whether there is enough data. We refer to Section 3.3 for concrete examples where the MDP is fully defined and a possibly remaining ambiguity stems from partial observability. Interested readers can also find more such examples in Lang et al. (2024).
In all examples, we have a feature set Ω = {wi, W2, 4lg, W4} whose meaning will vary in each scenario. We also have observations 01, 02, 03, 01 05, though the observation set S E is sometimes only a subset of {or, 02 03, 04, 05). We will make use of the fcature belief function : R 2 R4. We represent each c(o) row-vector with entries e(o); [e(o)] (wx) follows:
c(o)) = (1,0,0,1)
(02) = 1,0,)
E(03) = (0,0,1,1)
(7)
(04)) (0,3,0,2)
c(0s) 00,
In all examples, we assume E, V, and (and thus overall M = (2,,A,EVV)) to also be known, though wo do not make them explicit.
Example 2.18 (Simple Completeness). The goal is to create chuldren stories & EEE E that appeal to specific child Alice. We assume that it is known that children care about the presence or absence of eractly the following four features = {Wi, w2, wa, w
w1 Rule-breaking.
W2: Non-linear story-lelling.
Wa: Scariness.
w4t Moral lessons.
We show Alice four stories O = =f0,, 02 03, 01) S lion (7)):
that give rise to this feature belief function (ef. Equa-
0
0
1
1
0
1
E R + R -
(8)
0
1
1
3
0
2
15

---

Published in Transactions on Machine Learning Research (08/2025)
Conceptually, assume these are the true feature strengths, meaning that X(o) c(o) for all O, i.e., Alice perceives the "true" omount of the four features for each story. The matrir elements mean that or contains rule-breaking and moral lessons, but the story-telling is linear and the story is not seary. 0д is very non-linear and contains quite some moral lessons, but shows no rule-breaking or scariness. Since the rows of Equation (8) form basis of TR we oblain
Σ Zoc(o) Zo E RR OEO
and by Theorem 2.13 it follows that the model is complete. Thus, the ambiguity vanishes and we can infer the return function from Alice's s feedback Go on the stories in O (ef. Interpretation 2.14).
We demonstrate this now explicilly- Assume Rg is Alice's 's (a priori unknown) reward object, and that the resulting observation return function Go = ={Rp) E R is given by Alice's explicit feedback on the four stories 01...., 04 as follows:
Golo)) = -3 Go(oz) = Go(oa) = Go(os) = 4.
(9)
Representing Ry and Go column-vectorss this means:
1
0
0
0
1
1
Rn = E(Ro) = Go
0
0
1
0
0
3
0
2
4
This results in
0 1
0
1
33
1
3
0
1
3
2
0
0
1
1
0
-2
0
1
1
2
Rn
0
1
1
0
=
0
3
1
1
0
0
2
0
3
0
-1
Thas, we recovered Alice's 's reward object: She positively values mon-lincar story-telling (2) and scariness (1) but does not like moral lessons (-1) or rule-breaking (-2). Together with the knowledge of the ontology A to associate features to stories, we can create more enjoyable stories for Alice.
This ezample shows that knowledge of relurn-releuant features and a successful modeling of the feature belief function con go a long way determine the correct return function: We essentially need only four "data points" " (in the form of obsercation returns Go(o)) to determine G.
Example 2.19 (Completeness from symmetries). We assume the same situation as Example 2.18 except this time, we assume that we only have three observations O
for, 02, 01):
0 1
E C + RO, 8
0
1
(10)
0 2
Note that the lhird row now represents e(os) since 03 is not in O. Assume we have a priori information that children who like scariness do not like moral lessons and vice versa; i.e., it is known that Alice's 's reward object satisfies the following equation:
Ra(wa) = - RR(ws)-
The set of all reward funclions with this properly forms a vector subspace V C R It turns oul that this implies kE)(E) n V = {0). Indeed, let R2 E ker(E) n.. Then E(R,)) = implies Rg(w)) = R2((2) = Rylly) = 0, and Rg E implies R2(wz)) = RR,((w = 0. With ker(E) V 0, the ambiguity vanishes: Thus, the model is complete and the return function can be inferred from Alice's 's feedback Go O (01:02, 04) (ef. Interprelation 2. 14).
Amb A(ker(E) V)
16

---

Published in Transactions on Machine Learning Research (08/2025)
Example 2.20 (Ambiguity from undetected vulnerabilities). The gool is to correctly evaluate coding-agent to produce valid and safe code blocks & E E. We assume that exactly the following four features are relevonl. for evaluating code:
w1 Code- vadder-nnnerailltyy
W2: Syntar error.
wa: Simplicily.
War Validity.
We show the humman evaluator four code-blocks O = {0z, 03, 05)
C
=
that give rise to this feature belief
function (ef. Equation (7)):
1 0
8: R +R.
E
3 0
2
0
1
Crucially, we con assume that O5 does contain a code- mode-vulnerabbdtty, but the human does not detect it. In other words, the ontology E RD assigns the feature strengths X(o) = 0, & 00,0,0,11 1) = =(os). the human had detected the code-bulneraiilty, then os would be mathematically equivalent 0O in Example 2.18, the model would be complete, and the ambiguily would disappear.
obtain that kerlE) nv + {0} contains the reward object Ry E R with Ry(w1)= = and (y = 0 for all w # w1. With Rn being the unknown true reward object and Rn == Ra + Ra we then have E( ERR)) = E(Rg) = Go, and so we canmot distinguish between the reward object Ra and the true reward object Rg from the human's feedback. Let G ((Ra) be the true return function. Then the relurn function G A(aa) = A(Rg)) E FCM is compulibl and satisfies:
However, this is not the case. Assuming V R
G(E) G(E) + [A(RA)]
-G(E) + (A(E), Rn))
=G(E) + [A(E)]
This return function positively rewards code-vulnerabiitties and result from attempt to infer G from Go.
Example 2.21 (Rescuing the return inference). Consider Example 2.20, but with the code block or added to the observations, leading to all five observations O {01,...,08, .5) This results in the following feature belief function (ef. Equotion (7)):
0
0
1
1
0
Σ: R + R
-
0
0
1
0
3
0
2
0
0
1
Then ker(E) 0 and thus the ombiguity disappears: A A(ker()) = 0, leading to correct return function inference (ef. Interpretation 2.14). This example highlights that even when the human misinterprets some observations (in our example: 03), the correct return function can sometimes inferred as long as the human's feature beliefs over all observations have enough coverage.
3 Human belief model covering
So far, we assumed that the true belief model is known precisely, and studied when this allows recover the return function on trajectories from the human's feedback on observations. Knowing the true belief model
17

---

Published in Transactions on Machine Learning Research (08/2025)
might not be realistic, and so need to relax this condition. One possibility is to only require that we specify belief model that can cover (i.e., represent) all return functions and observation return functions that are represented by the true model. If it does, and if it is complete (meaning the ambiguity disappears), then we can intuitively use such a model for a correct return function inference. In Scction 3.1, we define this notion of belief model covering. We show that the ambiguity of covering model al least. as large as the ambiguity of the true model. If the ambiguity disappears, the covering model can be used to infer the correct return function from the human's feedback (Theorem 3.2).
In Section 3.2, then find an equivalent condition of belief model covering, based morphism between belief models. In many examples in this paper we have natural morphism that accounts for a model covering. In the same theorem, we also find sufficient condition for the existence of a morphism in terms a linear ontology translation from the covering model to the covered model that also respects feature beliefs (Theorem 3.5). If such an ontology translation exists, then the covering model has the capacity simultancously linearly represent the coverod model's concepts and beliefs.
notion of
In Section 3.3, study a detailed example of belief model covering: We consider human with ontology that is invariant under symmetry transformations in the environment, which implies that human's 's model with a model that assumes rymmetry-invariane reward functions. We also demonstrate that this covering model has a vanishing ambiguity, improving upon the ambiguity of a model that considers general reward functions. In Section 3.4, we conclude with a practical proposal for how to find a belief model that covers the true human model, based on using foundation models for the ontology and feature belief function. That the resulting belief model might cover the true human model is motivated by research sparse autoencoders (Cunningham et al., 2023; Bricken al., 2023), which will interpret linear ontology translations.
cover the
In this whole section, we assume an MDP together with trajectories E and observations O as given.
3.1 Human belief model covering and its implications
Definition 3.1 (Belief model covering). Let MM = (2,, and M (Ω, be two human belief models. Then we soy that M covers M if for all E V there erists 6 E D with A A(v) and E(t) E(v).
This means that M represent all return functions A()) and observation return functions E(v) that are represented by M. We can visualize this follows:
A(v) A(
R
A
YU
30
E
Ul
R
Ul
(11)
E
E(u)= = E())
RO
Theorem 3.2. Let M = (2, (0,A,E, and M (,, D) be two human belief models, and assume that M covers M. We think of M as the "true" human belief model representing G A(Rg) and Go E(Rn) with reward object Rn V. Then have:
1. M M
If M also covers M, then A Amb
3. There is an RS E D with E(Rg) =Go and Ā (RR)= G, and M also represents G and Go-
18

---

Published in Transactions on Machine Learning Research (08/2025)
4. Assume M is complete. Then every reward object R/ v with {(R)) Go also satisfies (R2))
G. In other words, the set of feedback compatible return functions is given by (GM) = (G)
D). To prove claim 1, let G' = = E Amb where R2g E ker(E) nV. V. Then by the definition of M covering M, there exists an R' VV with A(H)))) A(H))) G" and ( ((R) O. The latter implies R E ker(E) V, and $2
Proof. By Proposition 2.9, have Amb = ( V) and Amb (ker(E) n.)
12
so G = ((GG) E (kerr()) nv) Amb M This proves claim
Claim 2 then immediately follows from claim 1. Claim 3 is also immediate by the definition of M covering M and using that Go = E(Ra) and G (Ra).
Now we prove claim 4. Thus, assume M is complete and let R E 1 be reward object with E(R = Go. By claim 3, there exists an R E D with (()))= = Go and (R G. It follows R Ta-Ra Ra E ker (8) nn ker A) n D, where we use that M is complete in the last step. Consequently, we have
((R))) - G,
thus proving the claim.
In Theorem B.8 we present version of the preceding theorem for the case that feedback is given by choice probability function instead of Go. In Appendix C.4, we see several applications of the first two statements of the thcorem to determine inclusions and equalities of ambiguities.
We can interpret Theorem 3.2 as follows: Given a "true" model M, but using a covering model M, we lose something since the ambiguity of .M is possibly larger. However, M able represent the true return function and observation return function, and if the ambiguity of M disappears, then the set of return functions compatible with the human's feedback that can be inferred using M is exactly {G}. In other words, we can then infer G from the human's feedback and M without knowing the true human belief model M, thus relaxing the assumptions baked into Question 2.7. In Section 3.4 discuss hypotheses for finding covering belief model M in practice.
3.2 Morphisms human belief models and ontology translations
One drawback of the definition of a belief model covering that is hard to directly test: One would need to iterate over all observation return function as This analysis could be simplified if we could find one function $ R - R that maps each E V directly to e E V. The covering property would then translates to properties of such functions. We state these properties in the following definition:
to check whether there exists 0 E D that gives rise to the same return function and
Definition 3.3 (Morphism of human belief models). Let M = (, and M = E be human belief models. Then a linear function + RD - R is called a morphism of human belief models if the following holds:
1. +(V) D.
2. Aly = À o Φlv.
3. Ely ølv-
We write the morphism also as M - M.
The following visualization, an adaptation of Equation (11), makes intuitive that the existence of morphism is equivalent to belief model covering; we will prove this in Theorem 3.5:
19

---

Published in Transactions on Machine Learning Research (08/2025)
( (
R
A
R
0
((v)
E
Ul
Ul
R
We now work toward a sufficient condition for belief model morphisms that we call linear ontology trans- lations. For this relationship, recall the form x: - RR, c: O > R of the ontology and feature belief function of a belief model M = (Ω, A, E V), respectively. Recall from Equation (5) that for a reward ob- ject v V and trajectory E, the corresponding return given by [A(v)] viewpoint, property 2 of belief model morphisms + implies that for all v E V and & E E, we have
(A(E). Under this
(A(E), (())) ())
and similar for and ε. By defining properly of transpose matrices (also called adjoints); we obtain for all V and & E:
(()) (X(E..))
and thus क o = A. This equation means that interpret क as translation from the ontology Ā to the ontology A. We define:
Definition (Linear ontology translation). : RR., 6:0 R and x: E - 7: 0 R
be two pairs of ontology and a feature belief function. A linear function V R² RR with V X is called linear ontology translation from X to X. Furthermore, call it belief-compatible with ? and if we also have V o =
These notions are connected in the following Theorem:
Theorem 3.5. Let = (Ωt A, E,V) and M = E, V) be two human belief models. Let X, c, G,,,E be the allernalive represenlalions of the onlologies and feature belief funclions. Consider the following stolements:
M covers M.
There exists a morphism of belief models :: + M.
There exists function + R - R with Roo A and E
4. There is a function v R² + R that is a linear ontology translation from ā to that is also celief-compattble with ? and c.
Then 1 and 2 are equivalent, and and are equivalent and both imply and 2:
2
4
V from can be defined as v ΦT for & from 3. Vf = R then all four statements are equivalent. Finally, if2 holds and +DV) D, then M also covers M and A A
20

---

Published in Transactions on Machine Learning Research (08/2025)
Proof. The implication from the second to the first statement follows immediately from setting @ ((v) in the definition of belief model covering. For the other direction, consider the following diagram:
g
V
R = + R
In this diagram, we define f = (Alv, Ely) and = (Al E kp)) Then statement 1 is equivalent to im S im(.). By Proposition A.4, there thus a linear function Φ V D with go0= =0=
V
R =
+
R
Extend @ arbitrarily to linear function + R R with Ø/y Ø e-g. by extending basis on V to basis all of R? Then for all V, the diagram shows that we have
= ((((()))
Consequently, Aly = A 1/v and E/y = E k/v. Thus, t morphism of belief models.
That statements 3 and are equialent and v can be chosen as " follows from Proposition A.2. That 3 implies 2 is clear. That all statements equivalent if V R is also clear.
For the final statement, assume that 2 holds and that +(V) = D. Then for all T E & there is v E y with (v) C. Since & morphism, this results in
(v) ((*(v)) = A(
E(v) = E(l()) = E((),
showing that M covers M. Amb = Amb M then follows from statement 2 in Theorem 3.2.
In Appendix C, we show that human belief models, together with their morphisms, form category (Lane, 1998). In Appendix C.4 we then construct a large diagram of human belief models together with their natural morphisms, which then also allows for an analysis of their ambiguitics.
The preceding theorem shows that sufficient. condition for M - M to be a human belief model morphism is for Φ" R R to be a belief-compatible ontology translation: ΦT o --= X and Φ of = Intuitively, this means that our model M is "expressive" enough to allow for the true model's concepts and beliefs to be linearly represented. We will draw more connections to linearly represented beliefs and concepts in Section 3.4.
3.3
An example of symmetry-invariant features and reward functions
We now study an MDP with natural symmetries in the environment. We can then reasonably assume the human's ontology to invariant under these symmetries. We explain how one can cover the resulting human belief model with model that distinguishes between symmetry-related states, but compensates for by assuming that the valid reward functions are symmetry-invariant (van der Pol al., 2021). The ambiguity in this covering model will disappear, while the ambiguity of third model that allows for generic reward
21

---

Published in Transactions on Machine Learning Research (08/2025)
functions does not. This demonstrates the usefulness of both covering belief models and careful choice of the vector space of valid reward objects. In particular, the analysis shows that encoding a priori knowledge of symmetries into the human belief model can be fruitful for inferring the correct return function from the human's 's feedback.
We proceed by first defining the MDP, set of trajectories, and observations. Afterward, we define all three belief models together with their morphisms, demonstrating model coverage. Finally, we conclude with the ambiguity analysis.
3.3.1 Specification of the MDP, E, and O
Our MDP is a 2x2 gridworld with a movable hand and a fixed button B, inspired by the robot-hand example from Amodei et al. (2017). States look like this:
H
(12)
In total, the set of states S has sixteen elements, one for each combination of the position of H and B. The set of action is given by A= {L, R. U, D, P), where the first. four actions move the hand: L to the left, R to does not change the state, and is conceptually meant to "press"
the right, U upward, and D downward. the button if the hand and button are in the same position. a movement goes toward an adjacent wall the gridworld, then the state also does not change. This specifies the transition function T 8 A Po, the initial state distribution, is uniform distribution over the following four states:
H
H
B
B
B
B
H
H
(13)
The time horizon is = 3. The unknoun true return function given by
G(E)
Σ R(s t=
where R(s, at) = the hand H and button are in the same position 8, and if = P, and R(s,, a,) otherwise. In other words, the return function rewards pressing the button. This completely specifies the MDP AS T, Po. G).
The Trajectories E are given by all sequences of four states and three actions that start with state sampled from Po. and where cach transition is compatible with the description above. The obscrvations O of the human evaluator are given by views " "from below". We assume the human does not observe movement actions (but may be able to infer them the hand visibly changes position), but does observe whether the button that projects the view and removes movement information, we suggestively depict for an example trajectory & and its observation O(E) in this figure:
was pressed. Formally,
OEE) for surjective function 0
22

---

Published in Transactions on Machine Learning Research (08/2025)
H
R
H
D
P
B
B
HB
HB
P
H
HB
HB
HB
See Appendix D.1 for some mathematical details.
3.3.2 Three human belief models and their morphisms
Crucially, assume that the human evaluator does not stale-action pairs as features in the ontology, but instead representatives of ceammetrry-euivvlennee classes of state-action pairs. This is reasonable since we can a priori assume that the human evaluator does not care about the orientation of the scene. In other words, we consider the symmelry group = D4 of the square, which identifies two stale-action pairs if they are related by a rotation of 0%, .90", 180%, or 270", or a reflection along the vertical, horizontal, or one of the diagonal axes, This leads to just three representative states
$0
$1
$2
(14)
HB
H
Overall, the set of representative state-action pairs is given by S O A' with A Aa (T., P} and A = [I., U, D, P).² Let h : S x A + S x A map each state-action pair to the unique representative. Details on everything discussed so far can be found in Appendix D.2.
We define the human's ontology A E E RSA via
2 (s,)) = Σ (h(r,,())))))))
which is the number of times that, up to symmelry, the slate-action pair (s, a) appears in the trajectory 5. Interestingly, this ontology is invariant under transforming trajectories ç via symmetries since is in- variant under transforming state-action pairs. Set A R R as the linear function correponding to A via Proposition A.1, and let T RSXA + R be the function from Example 2.1 without discount- ing (> 1).3 Finally, let ht RSx + S be the function h (+t) = R h (see also the discussion surrounding Equation (1)). Then in Equation (24) we show
A = Tohh
(15)
Let b: A(=) C R be the human's trajectory belief function, where we define b(o) as the uniform distribution over all { E E that get observed as o: O(E) = (ef. Equation (25)). We then define the human's
feature belief function O R by
[c(o)] (s.
Σ [b(o)] (s.))
SEE
2sy and have fewer representative actions since L and U, and also R and D, related by the reflection along the diagonal axis from top left bottom right. This transformation leaves the state invariant and maps between the actions. Note the small difference that consider reward functions that only depend state-action pairs instead of action-state transitions.
23

---

Published in Transactions on Machine Learning Research (08/2025)
This is the ezpected number of times that, up to symmetry, the state-action pair (s, a) occurs in trajectory that led to observation o. Set & R + R and : R² - R as the linear functions correponding to and b via Proposition A.1. Then as consequence of Equation (15), we obtain
ع = =BoPoh"
(16)
we show in Equation (26).
Finally, we assume that we have a priori knowledge that the human's 's reward object lies in the subvectorspace V S I given by reward objects R with R(s a) = 0 whenever o P. In other words, we know that only the pressing-action can be rewarded, but we do not know a priori that it is only rewarded when the hand is over the button. Furthermore, define V" _ RSx likewise as reward functions with R(s, a) = 0 whenever a o P. Consider the commutative diagram
R
RšXÁ Ul V
R
1dasKA
R Ul
h" (V))
V
Bor
Ro
B I ah""
This establishes three human belief models
Mi = (S A, roh", BoToh", V),
M2 = (S A, r, Bor, I²())),
M3 = (8 A, r, Bor, V*),
together with the morphisms
A- M2 idgaxA M3. Mi
Our aim will be to show that the ambiguities of Mj and M2 will vanish since these models leverage priori knowledge of symmetries, while the ambiguity of Ms will not vanish. Intuitively, Mi is the true belief model of a human evaluator with symmeery-invvriint features. M2 is the model that we "specify" and with which we "cover" the true belief model. That it indeed covers M, follows from Theorem 3.5 and the existence of the morphism h² M1 MM+MM M2-
Write g-(s,a) a) for the action of tymmmetrytraansfomattio g E G = Da of the square a state-action pair (s,a) a) (ef Appendix D.2). Then the set of valid reward objects of M2 is given by
h (²()) R Vg G: R(g a)) R(s,a) a) and Va + P: R(s,a) = =]]-
(17)
In other words, it is the set of symmelry--nnariinn reward functions that don't reward actions unequal to P. Such reward functions play role in caametry-invariian reinforcement learning (van der Pol et al., 2021). Finally, M3 is the same model, but with larger set of valid reward functions that are not necessarily symmetry-invariant.
Mathematically, all three models are faithful by Proposition 2.16. As an aside, they are also balanced (Definition B.2) by Lemma B.3, which essentially means that the ontology and feature belief functions are row row-constant. The three models and their morphisms are also closely related the three models MF. MSXAx8 and Sxxxx that we study in Appendix C.4.
24

---

Published in Transactions on Machine Learning Research (08/2025)
3.3.3 The ambiguity analysis
We now analyze the ambiguities of the three models M2, and M3. Since the morphism h" maps the set of valid reward objects V of Mı precisely to the valid reward objects h h²(V) of M2, by Theorem 3.5, Mi and M2 have the same ambiguity. Thus, let us analyze the ambiguities of Mi and My.
For showing that it vanishes, we simply show ker I ) n ker(E) - O. Thus, let R RšxA be reward object in V with ( O. We need to show R = O. Recall the representative states so, $1,82 from Equation (14). Since R E V, we have R(s.a) a) = 0 for all E {s0, 81, and all + P. and so we simply need to show R(s;, P) = 0 for all = 0, 1,2. 2. Consider the following three observations:
For M1, Proposition 2.9 shows that the ambiguity is given by (Ich") [
P
00
H
B
B
HB
HB
P
P
H
B
H
B
H
B
H
B
P
P
0
H
B
B
>
H B
B
(02) state-act ion pair that is compatible with the observation 02- Then, [E( (or) = 0 implies R((1, P) = 0 since P) is the only state-action pair other than (52, P) that is compatible with the observation or and could a priori have a non-zero contribution to the reward. Finally, [&(]]] (oo) = implies R(so, P) = for similar reasons. We present details of these arguments Appendix D.3. Overall, we have thus showed that R = 0, and thus ker(E) n== 0, which implies Amb A(ker(E) n)) 0 (ef. Proposition 2.9). Since Mi and M2 have the same ambiguities, also our covering model M2 has vanishing ambiguity: Amb 0.
Then casy to show that [E(R)] (o2) = 0 implics R(s2, P) since (s2, P) is, up symmetry, the only
Now we show that the ambiguity of M3 does not vanish. Consider the following two states, which are synmetry-transfommdd versions state 81 from Equation (14):
H
B
$1
Sf
B
H
Then, let : x A > IR be the reward function with
1. = s and = P
R((s.a)
-1 s of and P
0less
Clearly, have R' E V. Then note that for all observations o, have oBo (s) symmetry reasons.4 Thus, we have ()R) (o) = 0 for all o E O, as we detail in Equation (27). This shows 0 & R' E ker(B r)ny, and consequently 0 0f T(R') E [ ())v") As (Proposition 2.9), proving the claim that the ambiguity is nontrivial. Crucially, in order to construct R', we needed to allow that the symmedry-related state-action pairs (s/, P) and (s", have different rewards.
(Bo () for
For showing this, recall that (BoT)o,(a.)) simply the expected number of times that statc-action pair (s, a) appears trajectory that gives rise observation For each trajectory, the up-down mirrored trajectory creates the observation. leading aforementioned symmetry.
25

---

Published in Transactions on Machine Learning Research (08/2025)
3.3.4 Conclusion of the example
This example highlights that priori knowledge of symmetry-invariant reward functions (via model Ma with its ambiguity h" (V),, see Equation (17)) or symmetry-invariant features (via model M1) can help to infer the correct return function from the human's feedback. We highlight again that one could in practice work with model M2 even if M₁ were the "true" model, the reason being that M2 covers M1 and has vanishing ambiguity. Future work could analyze this in more detail, by developing more general theory of symmetry-invariant human belief models.
3.4 A proposal for belief model covering in practice
So far, our discussion has been purely theoretical: We showed that if model M = (Ω, A,, is complete and covers the true belief model M = A, V), then the true return function G can can be inferred from the human's feedback Go (Theorem 3.2, statement 4), This raises the following two questions:
How can M be specified?
2. How can G be determined in practice, using M and Go?
We now give preliminary answers to these questions in Sections 3.4.1 and 3.4.2, based using foundation models for both the ontology A and the feature belief function E. We hope our idcas can inspire future empirical work.
3.4.1 Defining M for answering question 1
To answer question first, one needs to choose an MDP together with trajectories E, and observations O. Ideally, whole trajectories & E = or parts of them, and all observations, can be "tokenized" so that one can feed them into foundation models. Let Ā + R² be a foundation model, which we interpret as a function from trajectories to internal representation space with many neurons." 5 Then, define Å R - R the linear function corresponding to according Proposition A.1.
For À to be valid ontology that part of a belief model that covers the true model, by Theorem 3.5, we need there to exist an (implicit) linear ontology translation V R²² - Rº. There is substantial prior work showing that human concepts are represented linearly in foundation model's representation spaces (Mikolov el al., 2013; Park et al., 2024b; Turner et al., 2024; Nanda et al., 2023; Wang et al., 2023; Gurnee & Tegmark, 2024). Most relevant to our claims, sparse autoencoders (Cunningham directly construct a linear transformation that maps from foundation model's representation space to a space of human-interpretable features, thus constructing function akin to our (only implicitly needed) linear ontology translation V.
al., 2023; Bricken ct al., 2023)
Notably, Ā needs to be a capable foundation model for such a linear function to have any hope to be an exact ontology translation. After all, imagine we show X the Riemann hypothesis: then it cannot determine whether this hypothesis is true, which we consider an important feature that likely appears in the human's ontology X. Since we are concerned with scalable oversight, which about the problem of ensuring alignment of future, powerful AI systems, we assume À to be very capable, and thus that V is exact ontology translation. Overall, this gives some preliminary confidence that for capable foundation model Â: : + R², there will exist linear ontology translation V Rº + Rº, such that V Vo X = X.
is not vastly superhuman,
Now, let 7: O - RD be another foundation model (in the proposals below will be an adaptation of 2). Then, define E. R + RR the linear functions corresponding to ē according to Proposition A. Set
D RR Set M (,)
5This means that we remove the output functionality from this model.
26

---

Published in Transactions on Machine Learning Research (08/2025)
Why when would this be useful specification? As discussed before, we need M to be complete and to cover the (implicit) "true" belief model M. Based on sufficient conditions we found in Theorem 3.5 and Theorem 2.13, we thus need to ensure the following two properties:
(i) The ontology translation V : + R is belief-respecting: =
(ii) For all E X(E) is contained in the span of the image of ē X(5 E { EEEEoZoe( Zo E ER..
Ensuring (i). This is the most speculative part of our proposal. Crucially, if the human does not recoguize the presence of a feature in O due to limited capabilities, then è should ideally also not recognize this feature so that translating from one ontology into the other respects beliefs. Conceptually, this means that ē should simulate, in the feature space Qt the human's beliefs and understanding. We make three proposals:
For research prototyping, one could restrict to a problem with "pure partial observability". In other words, choose a sctting where observations are given as o= O(g) for trajectorics & E E, and where potentially superhuman capabilities do not make it easier to infer further return-relevant aspects of g from O(E). This is intuitively the case if information is simply entirely "missing" from observations. Then, simply choose ē ;= X, applicd to observations instead of trajectorics. In this case, the fact that crucial information is equally obstructed to the human with feature belief function e and to the AI with 2 should ensure that the ontology translation also correctly translates feature beliefs: oo?== c.
Now consider a setting that may go beyond "pure partial observability". One speculative idea for how construct ? is to prepend "belief prompt" bp to inputs of À that nudges the model to think more in the way a human evaluator would think (Park et al., 2024a):
ē(o) = (op = X(bp,o).
An example of such a prompt would be
bp = "Think about the following input like a typical human evaluator:"
Alternatively, one could potentially achieve this by finetuning to obtain غ. Unfortunately, while foundation models can simulate the behavior of specific people in their outputs, unclear whether this also reflects in their internal representations. For example, prior work shows that the truth- value of statements can sometimes be linearly predicted from internal representations even when the model lies, leading to the potential for AT lie detectors (Azaria & Mitchell, 2023; Burns et al., 2023b). However, other work trains foundation model to predict human behavior in experiments and finds it to have internal representations that can predict human neural activity when engaging in the same task (Binz et al., 2024).
Alternatively, one could also consider defining ट = Rearly an earlier training checkpoint of Being an carlier training checkpoint, à would then be less capable than 2, leading to the potential that has the same blindspots in understanding observations
O as the human evaluator.
Ensuring (ii). To ensure that for all & E E we have A(E)= E {Eoso ZoZ(0) Zo B). it is important to
ensure that the image of ē: O - RD is "large", i.c., spans as much as possible of the representation space. To form more intuitions on this, note that by Theorem 3.2, M being complete (which would be implied by property (ii)) requires also the true belief model M to be complete. Again, by Theorem 2.13, a sufficient condition for this that for all E E, X(E) E { EoEo Zoc(o) Zo E RR. In particular, any "bad" feature W E Ω that can ever be present in a trajectory (meaning [w()) * needs to also sometimes be believed to be present by the human (meaning there exists an E O with [e(o)] (w) + 0). This is relaxation from the requirement that the human understands all observations perfectly, but does mean that there needs to be a variety of observations o E O that the human understands well enough to detect their diverse underlying problems. It will depend on the specific MDP and setup of an experiment to reason about how to ensure or purposefully violate this property.
27

---

Published in Transactions on Machine Learning Research (08/2025)
3.4.2 Learning G using MM and Go for answering question 2
Now that we have discussed preliminary approaches for how to specify a complete model M = (2, D) that covers the true belief model M, we can turn to the question of how to use M and the human's 's feedback Go : O + R to determine the true return function G. By statement 4 in Theorem 3.2, we can determine G = Goo
as G (Rg)) for R E R with )
We now unpack that. Remembering the relation between A and the foundation model I, and E and the foundation model ? via Proposition A.1, we want to determine R such that for all o E O, we have
Go(o) (E( (a(o), (E(O),
(18)
This can be achieved by attaching Ri as a linear reward probe to the representation space of ē and learning the function Go by supervised learning, with all parameters except the reward head being frozen.
In the case that Go cannot be directly evaluated but that it is indirectly accessible in the form of choice probabilities between observations, one can learn R by logistic regression as in Christiano et al. (2017). See Appendix B.2 for a possible correspondence between E(Rg) and the resulting choice probabilities. No- tably, the approach via logistic regression, however, loses a theoretical guarantee: Namely, Go can at best be learned up to an additive constant. If and À were balanced (meaning that the total weight of feature strengths is constant over all observations and trajectories, respectively), this would lead to the inferred G also being correct up to an additive constant by Proposition B.6. Since additíve constants in return func- tions are inconsequential for policy optimization, this would be fine. However, typically the representation spaces of foundation models are not normalized, the balancedness property does not hold, and this guarantee breaks. It is then an empirical question to what extent this breakage is an issue or how to resolve it.
After successful learning, we can then compute the true return function G for & E E as:
G( = [A(RR)](E) = (A(E).
(19)
In other words, we atlach the learned linear reward probe R2 to the representation space of Ā and use it to compute returns. These can then be used to train a policy to maximize the policy evaluation function Equa- tion (4) via standard reinforcement learning techniques.
3.4.3 Further remarks
Looking at the definition of the human belief model M, we see that it includes the ontology A R² - R and feature belief function E. RD + R These can be extremely large matrices. However, since in the process of training Rg and computing G, we only need to be able to query the resulting observation return function and return function on specific observations and trajectories in Equations (18) and (19), the matrices never need to be stored or used in their entirety. Thus, the size of the matrices is not a concern.
We also remark on a special case we mentioned before: If we are in a case of "purc" partial obsorvability where observations o E O contain no information that À understands better than the human, then we proposed to simply set. := X, applied to observations o E O. In that case, the procedure we describe is essentially classical RLHFF, with the only crucial difference that during training of R3. we only show observations, instead of full trajectories, to the reward model. This prevents model misspecification where the data. the model reads differs from what the human sees, and could theoretically allow generalization to data & E E. Lang et al. (2024) consider naive RLHF, where the initialized return function reads entire trajectories during training time while the human's 's view is obstructed, leading to issues of deceptive inflation and overjustification after policy optimization.
3.4.4 Limitations for the proposal
We conclude by summarizing the limitations of the proposal in its current form. At the very foundations, it assumes that real humans are compatible with suitable belief model as defined in this work, that human
28

---

Published in Transactions on Machine Learning Research (08/2025)
values are captured by return functions over trajectories, and that the model allows for linear decomposition of the return function over features. All of these assumptions are speculative and may need refinement.
The proposal then assumes that M is an ezact covering belief model for the latent human belief model. In particular, it is assumed that À reads entire trajectories and allows for the existence of a linear ontology translation to the true human ontology. Finally, the feature belief function ? needs to "translate" to the true human belief, and it is unclear whether our proposals can achieve this sufficiently. To achieve completeness, the proposal also requires careful environment design to ensure that the beliefs ?(o) "cover" the feature combinations X(E) sufficiently.
In conclusion, substantial future work is needed to validate the utility of the proposal. We propose such work in Section 4.3 for many of the limitations above.
4 Discussion
In this discussion, we summarize our work, survey related work, propose ideas for future work, and conclude.
4.1
Summary
In this work, we have introduced the notion of a human belief model, based on modeling a human's ontology and feature belief function. The goal of such a model is to aid the inference of the human's implicit return function from feedback. In our framework, the feedback, in the form of an observation return function, is viewed as carrying information about the reward of features that the human believes to be associated with an observation. Once the feature rewards, in the form of a reward object within a valid set, are inferred, they can be used together with the human's ontology to infer a return function. We defined and characterized the resulting ambiguity the return function in terms of the belief model and showed that for complete models, the ambignity disappears. Complete models have an important sufficient condition in terms of a linear coverage of the features of any trajectory by feature beliefs of observations (Theorem 2.13). This shows that for observations that are varied enough and provide ample information in total, correct return function inference is possible, which we demonstrated in simple conceptual examples in Section 2.9.
Since the human belief model is not known in practice, we then introduced the notion of belief model coverage. Here, a model M covers another model M if M can represent all return functions and observation return functions that can be expressed in M. Tf a complete model covers the true belief model, then can be used for the return function inference just as well (Theorem 3.2). We then characterized belief model coverage in terms of belief model morphismss, and found an important sufficient condition given by the existence of linear translation from the covering model's ontology to the true model's ontology that is also compatible with the feature belief functions (Theorem 3.5). We then studied a conceptual example of a human with symmetry-invariinn feature beliefs, which we could cover with a model whose completeness stems from the valid reward functions being symmetry-invariant invariant.
Finally, in Scction 3.4 we sketched a proposal for how to find covering human belief models in practice, by using foundation models for both the human's ontology and feature belief function. For the latter, it is important to ensure that the foundation model has a similar understanding of the observtions the human evaluator, for which we sketched out three proposals. That the resulting belief model might cover the true belief model relies on prior research on the linear reprosentation hypothesis, which provides evidence that a belief-repectting linear ontology translation could indeed exist. 6 Our hope is that our proposal can help to find covering models that are casier to determine than the return function itself, which might subsequently be learned with modest effort as an approach to scalable oversight (cf. Section 2.5).
4.2
Related work
Other human modeling approaches. The human belief models we introduce in this work are largely about modeling human's ontology and feature belief function for learning from the human's feedback.
We emphasize again that our theory only requires its existence and no explicit specification this ontology translation.
29

---

Published in Transactions on Machine Learning Research (08/2025)
Similar work (Marklund & Roy, 2024) is about the special case of humans observing partial trajectorics and forming beliefs over the rest of the trajectory.
There is also work that models aspects about humans different from their beliefs. For example, reward- rational choice (Jeon et al., 2020) requires modeling human choice probabilities for choices over various options like trajectory-pairs, language-uttteracee, or initial environment states. An example of this frame- work is inverse reinforcement learning (Ng et al., 2000), which has also been considered in partially observable environments (Choi & Kim, 2011). Reward-rational choice can be regarded special case of assistance prob- lems (Fern et al., 2014; Manfflld-Menelll et al., 2016; Shah et al,, 2021), which requires model of the human's action selection in cooperative two-player game. This framework has recently been generalized to a par- tially observable setting (Emmons et al., 2024). Much of this work makes specific assumpioos about the human's model, e.g. by assuming a Boltzmann-ratione or optimal selection of choices. Zhi-Xuan et al. (2024) instantiates a version of assistance games in which a human's utterance is modeled using a language model. Finally, Hatgis-Kessell et al. (2025) researches how to influence human evaluators to conform with a theoretical model of human choices. Compared to all this work, we model human beliefs about AT behavior instead of human actions or choices.? 7 In Section 4.3.1 we propose research directions to combine these types of work.
Deception in AI. Our work generalizes the human belief modeling from Lang et al. (2024), which is meant to address deceptive AI behavior that also surfaces in recent empirical work for cases where humans lack evaluative capacity (Cloud et al., 2024; Denison et. al., 2024; Wen et al., 2024; Williams et al., 2024). Deception in AI systems can also occur for various other reasons (Park et al., 2024c), e-g., when language agents are put under pressure (Scheurer et al., 2023). An important theoretical concern is deceptive align- ment, in which an AI system follows the given goals while actually planning later takeover (Hubinger et al., 2019). Recent work (Greenblatt et al., 2024) substantiates this concern by showing that the language model Claude plays along with new harmful goal for the purpose of preventing that the learning process updates its safety dehaviorin in the long-term. Finally, deception has also been formalized for structural causal games (Ward et al., 2023).
Surfacing latent knowledge. In Section 3.4, we already mentioned sparse autoencoders (Cunningham et al., 2023; Bricken et al., 2023), which construct an explicit linear transformation from a foundation model's reprosentation space to numan-interprotablee features, which is in the spirit of a linear ontology translation (Definition 3.4). Instead of decoding the AT's enföre ontology in a human-interpretable way, other paradigms seek to construct a reporter that can be queried for specific information (Christiano et al., 2021). In this direction, recent work builds toward AI lie detectors by finding internal linear representations of truth (Burns et al., 2023b; Azaria & Mitchell, 2023; Marks & Tegmark, 2024), with alternative interpretations of such findings discussed in Liu et al. (2023). Other work linearly predicts concepts like harmfulness (Zou et al., 2024), theft advice (Roger, 2023), the activation of a harmful backdoor (MacDiarmid et al., 2024), and concepts related to honesty and power, among others (Zou et al., 2023).
Amplified oversight. While our work aims to learn from leedback of humans who potentially lack capabil- ities, work on amplified oversight tries to increase the human's evaluation capabilities through AI assistance to achieve scalable oversight. Recursive reward modeling the general proposal to train AT models by reward modcling and then using their assistance to evaluate the next generation of AIs (Leike et al., 2018). Saunders et al. (2022) shows that model-written critiques of summaries can help humans find flaws that they would have missed on their own. This raises the question why to trust the critiquing AI, which leads to the idca to, in turn, criticize the critic. Recursively, this Icads to a debate in which the debaters are trained to produce arguments that are persuasive to a human judge (Irving et al., 2018). This requires for debates to surface true and useful information to the human judges, which has found support for reading comprehension tasks (Michael et al., 2023). Optimizing debaters to be persuasive then increases the human's 's ability to identify the truth (Khan et al., 2024; Kenton et al., 2024). Finally, some work uses AI to directly give feedback based on constitution (Bai et al., 2022) or model spec (Guan et al., 2025), thus removing humans from the evaluation process.
7Only in Appendix B do we consider human choices.
30

---

Published in Transactions on Machine Learning Research (08/2025)
Easy-to-hard and weak-to-strong generalization. Instead of amplifying the evaluator capabilities, other work for scalable oversight relies on easy examples that humans can reliably evaluate, which then requires the reward model to generalize to data that is harder to evaluate (Sun et al., 2024). Language models have also shown to generalize tasks like STEM questions from easy to hard data (Hase et al., 2024). reak-lo-slrong generalization differs by trying lo generalize from weak evaluation on potentially hard data (Burns et al., 2023a). Our proposal from Section 3.4 can be considered an approach to weak-to- strong generalization since we aim to learn correct return function G from feedback of an evaluator with potentially faulty beliefs. Since weak supervision is plausibly cheaper to obtain than strong supervision, there is also work that investigates tradeoffss to find the correct allocation of a fixed budget to label data with different data labeling qualities (Mallen & Belrose, 2024).
In Appendix E, we briefly interpret amplified oversight, easy-to- easy-to-hard generalization, and classical RLHF together with weak-to-strong generalization in terms of our theoretical framework by interpreting their underlying evaluator belief modcling assumptions and reasoning about their ambiguitics and learned return functions.
4.3 Future work
4.3.1
Theory extensions
Several extensions and gencralizations of our theory could be studied in future work. We assumed that we can exactly specify a belief model that covers the true human belief model. Future work could develop an approximate theory, where there is a quantifiable error in the belief model. This could proceed along similar lines as Lang ct al. (2024, Theorem 5.4), where the belief matrix is perturbed by a known error, leading to a quantifiable error in the inferred return function. Furthermore, we assumed that the human's 's return function is linear in the features of trajectories. One could study non-linear models, which would also theoretically ground the use of non-linear reward probes instead of the linear probes we propose in Section 3.4. We also assumed that learned return functions can read entire trajectories, which might be unrealistic for very complex environments. It would thus be interesting to develop a theory based on a second set of observations O' for the learned return function and the trained policy.. In the practical proposal, this would also mean that we cannot assume to have a foundation model X that translates to the human's 's true ontology instead, it would represent another feature belief function. Relaxing the capabilities of the foundation model could also help to model case in which the human evaluator has information that is hidden from the learned return function resulting policy.
The ambiguity is a measure of the information that is available in the feedback, given a belief model, for determining the human's return function. Future work could theoretically study concrete learning proce- dures, which are about extracting said information. One could, for example, study training distributions over the observations O to determine sample complexity bounds for the error of the resulting return function, or the regret of the resulting policy, This is theoretically interesting since the usage of the learned return function in policy optimization involves two shifts compared to the return function's learning procoss: First, it needs to evaluate trajectories instead of observations, making use of an ontology instead of a feature belief function; and second, the policy optimization leads to further distribution shift over trajectories, which can in turn lead to increased regret even if the return function seemed accurate before (Fluri et al., 2024). Finally, if there is remaining ambiguity, it would also be desirabe to study protocols for choosing a return function within the ambiguity, possibly using a priori knowledge about "human-like" return functions that is not captured by our notion of valid reward objects.
Going beyond our framework, one could attempt a synthesis with work that models the human's action selection, which we discussed at the start of Section 4.2. For example, could consider reward-rational choice or assistance games under the assumption that humans form beliefs based on obscrvations. Emmons et al. (2024) does a first step in this direction by assuming humans form rational beliefs based on knowledge of a prior of the agent's policy. In that framework, they then study the notion of information interference, which leads to an increase the human's uncertainty about the world state. If future work were to study more general, possibly faulty, belief models for humans in partially observable assistance games, this could make it possible to go beyond information interference to study deception.
31

---

Published in Transactions on Machine Learning Research (08/2025)
4.3.2 Empirical work
We would be interested in attempts to instantiate our proposal from Section 3.4. One could test the proposal in settings with synthetic humans with known ontology and feature belief function, which can for small MDPs allow compute the ambiguity explicitly. This should make it possible to make concrete predictions about experimental results. It would also be interesting to study settings with partial obscrvability in which capable AI does nol have an advantage over humans in understanding the meaning of the observations. As we discussed in our proposal, that should allow to use the same foundation model for the ontology and feature belief function, with the latter only applied to observations. This could then be compared with a baseline of "naive" RLHF in which the initialized return function reads entire trajectories during the learning process, similar to the conceptual examples from Lang et al. (2024). It would be interesting to study different levels of observability to dial the ambiguity up or down. We also encourage to break some of our theoretical assumptions, e.g., by using non-lincar reward probes. Finally, it would be desirable to learn how our approach can be combined with approaches in the direction of amplified oversight, easy-to- aas- generalization, or weak-to-strong generalization discussed in Section 4.2. For example, combination of our work with easy-to-hard generalization could seek to only train the reward probe on data where the human beliefs are modeled correctly by é, which should be a superset of easy data.
Instead of studying the whole pipeline, one could also empirically assess the underlying assumptions. The most precarious assumption is that it is possible construct feature belief function é, for which we discuss proposals in Section 3.4.1. In situations that are not purely based partial observability, we proposed earlier training
to prompt or finetune the model to step "into the shoes" of human evaluator, checkpoints of a model to decrease its capabilities to that of a human evaluator (assuming future models that would otherwise be superhuman). For this to work, there needs to be a linear ontology translation that is compatible with è and the human's true feature belief function e. One could test this empirically by using sparse autoencoders (Cunningham et al., 2023; Bricken et al., 2023) trained on an unobstructed capable foundation model, and evaluating the resulting human-readable features when applied to é.
4.4 Conclusion
In this work we ihelretically showed that if we could model human beliefs about AT behavior, then under certain conditions, this could help for the correct inference of human goals from human foedback. Since the theory applies even in cases where the human's beliefs are erroneous and based on partial observations, it is relevant to setting of scalable oversight where the AI is more capable than the human overseers. We hope that future work will build on our theory, empirically study our practical proposal, or engage in other research on how to make Al systems safe and aligned with the goal of reducing the risks of advanced AL.
Author contributions
Leon Lang developed the ideas, derived all theoretical results, and wrote the paper. Patrick Forré gave general guidance and feedback.
Acknowledgments and disclosure of funding
We thank Scott Fammons, Davis Foote, Erik Jenner, Micah Carroll, and Alex Cloud for discussions that shaped how we think about human evaluators with limited capabilities. We thank Open Philanthropy for financial support.
32

---

Published in Transactions on Machine Learning Research (08/2025)
References
Picter Abbeel and Andrew Y. Ng. Apprenticeship Learning via Inverse Reinforcement Learning. In Pro- ceedings of the Twenty-Ping International Conference on Machine Learning, ICML '04, pp. 1, New York, NY, USA, 2004. Association for Computing Machinery. ISBN 1581138385. doi: URL httpp:://dp...ooo org/ 10. 1145/1011330 1015430.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Conerete Problems AT Safety, 2016. URL https:
wwwrg/abs/1666... 1656.
Dario Amodei, Paul Christiano, and Alex Ray. Learning from human preferences. https://openai.com. Breearchleearnnn-Eae form--ooommmnnnn www.roooouaaa-aeeeeeeeeeeee preferences, 2017. Accessed: 2023-12-13.
Anthropic. Introducing Claude. http:p////wk...nnttoa.ccccco///r
2023a.
Accessed: 2023-09-05.
Anthropic. Claude's Constitution. http:///wwaaattpppoco/////////aa 2023b.
Accessed: 2023-09-05.
Amos Azaria and Tom Mitchell. The Internal State of an LLM Knows When It's Lying, 2023. URL argg/bbs/3344 13734.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, 2022.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamic Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Jaseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: Harmlessness from AI Feedback. arXiv e-prints, art 2rXiv.221..08073. December 2022. doi: 10.48550/ar 2212.08073.
Marcel Binz, Elif Akata, Matthias Bethge, Franziska Brändle, Fred Callaway, Julian Coda-Forno, Peter Dayan, Can Demircan, Maria K. Eckstein, Noémi Éltetö, Thomas L. Griffiths, Susanne Haridi, Akshay K. Jagadish, Li Ji.An, Alexander Kipnis, Sreejan Kumar, Tobias Ludwig, Marvin Mathony, Marcelo Mattar, Alireza Modirshanechi, Surabhi S. Nath, Joshua C. Peterson, Milena Rmus, Evan M. Russek, Tankrod Saanun, Natalia Scharfenberg, Johannes A. Schubert, Luca M. Schulze Buschoff, Nishad Singhi, Xin Sui, Mirko Thalmann, Fabian Theis, Vuong Truong, Vishaal Udandarao, Konstantinos Voudouris, Robert Wilson, Kristin Witte, Shuchen Wu, Dirk Wulff, Huadong Xiong, and Eric Schulz. Centaur: a foundation model of human cognition, 2024. URL https: : /ttp///axvvvv//////////e/e/aaavl org/ abs/2410. 20268.
Michael Bowling, John D. Martin, David Abel, and Will Dabney. Settling the Reward Hypothesis, 2022.
Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilé Lukoštūté, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christo- pher Olah, Danicla Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Liane Lovit,, Nelson Elhage, Nicholas Schiefer, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Robin Larson, Sam McCandlish, Sandipan Kundu, Scott Johnston, Shanna Kravec, Sheer El Showk, Stanislav Fort, Timothy Telloon- Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatdicld-Dodds, Ben Mann, and Jared Kaplan. Measuring Progress Scalable Oversight for Large Language Models, 2022.
33

---

Published in Transactions on Machine Learning Research (08/2025)
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards Monose- manticity: Decomposing Language Models With Dictionary Learning. Trunsformer Circuils Thread, 2023.
httpss//rransfrnnerocrcntsssspp 200023/mmooemmantfffftsssss /iads/x.ddxxmmil
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffct, Manas Joglckar, Jan Leike, Πlya Sutskever, and Jeff Wu. Weak-to- Strong Generalization: Eliciting Strong Capabilities With Weak Supervision, 2023a. URL https:
wwwpxvv.org/bbb//11... 09390.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering Latent Knowledge in Language Models Without Supervision. In The Eleventh International Conference on Learning Representations, 2023b. URL https://openreviie net had//toorurFidderrcubbbbbcee.
Jaedeug Choi and Kee-Eung Kim. Inverse reinforcement learning in partially observable environments. Journal of Machine Learning Research, 12(21):691-730, 2011. URL http://jmlr wtg/papers/vv2//chr choi ltmml.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Re- inforcement Learning from Human Preferences. arXiv e-prints, art. arXiv:1706.03741, June 2017. doi: 10.48550/ 10.48550/arXiv.111 arXiv.1706.0374.
Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts, 2018. URL httpp/////xxvva/r///aa///
Paul Christiano, Ajeya Cotra, and Mark Xu. Eliciting Latent Knowledge. https://docs gow.ll...oooo...oooooooooooo.... com/do edit, 2021. Accessed: 2023-04-25.
Alex Cloud, Jacob Goldman-Wetzler, Evžen Wybitul, Joseph Miller, and Alexander Matt Turner. Gradient Routing: Masking Gradients to Localize Computation in Neural Networks, 2024. URL https://arm org/an abs/2410.0...... 04332.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse Autoencoders Find Highly Interpretable Features in Language Models, 2023. URL https https://arxivv wwwp:/wwww.rg/a///////0/ 08600.
Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, Buck Shlegeris, Samuel Bowman, Ethan Perez, and Evan Hubinger. Sycophancy to Subterfuge: Investigating Reward-Tamperri in Large Language Models, 2024. URL https://arxiv.org: 240.///4401.06..
Mucong Ding, Chenghao Deng, Jocelyn Choo, Zichu Wu, Aakriti Agrawal, Avi Schwarzschild, Tianyi Zhou, Tom Goldstein, John Langford, Anima Anandkumar, and Furong Huang. Easy2Hard-Bench Standardized Difficulty Labels for Profiling LLM Performance and Generalization, 2024. URL https://arra org/ab s/2409. 18433.
Scott Emmons, Caspar Oesterheld, Vincent Conitzer, and Stuart Russell. Observation Interference in Par- tially Observable Assistance Games, 2024. URL https://arxiv: org/ /t////b///222299999
Alan Fern, Sriraam Natarajan, Kshitij Judah, and Prasad Tadepalli. A Decision-Theoreet Theoretic Model of Assis- tance. J. Artif. Int. Res., 50(1):71-104, may 2014. ISSN 1076-9757
Lukas Fluri, Leon Lang, Alessandro Abate, Patrick Forré, David Krueger, and Joar Skalse. The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret, 2024. URL https://arxiv. http://arrii..oo//bb//////// htgg/bb//2406.... 12/.///............
Google Gemini Team. Gemini: Family of Highly Capable Multimodal Models. tttps://storaget .eporrt.pfff pdf, 2023. Accessed: 0023-12-11.
leapis. coom//eepmidd-eeeeeeeeeeeee httpaammiiinnddddaaa///i///iaaaa
34

---

Published in Transactions on Machine Learning Research (08/2025)
P. J. Gmytrasiewicz and P. Doshi. A Framework for Sequential Planning in Multi-Agent Settings. Journal of Artificial Intelligence Research, 24:49-79, -9, July 2005. ISSN 1076-9757. doi: http://dx.doi... 111.:/663aaa......
URL
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Damny Livshits, Danny Wyatt, David Fsiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia- Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzman, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Ilyan Zarov, Imanol Arricta Ibarra, Tsabel Kloumann, Ishan Misra, Ivan Evti- mov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jehmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kemneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelcbi, Patrick Alrassy, Pengchuan Zhang, Peng- wei Li, Petar Vasic, Peter Weng, Prajiwal Bhargava, Pratik Dubal, Praveen Krishman, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Sil- veira Cabral, Robert Stojnic, Roberta Raileamu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hos- seini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Scohyun Sonia. Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Syd- ney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Uijwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ra- manathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaci, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharic Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arka- bandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisemman, Azadeh Yazdan Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mcjia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cymthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Ed- ward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presani, Emily Hahn, Emily Wood, Eric- Te, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian,
35

---

Published in Transactions on Machine Learning Research (08/2025)
Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Scide, Gabriela Medina Flo- rez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habecb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Tbrahim Damlaj, Igor Molybog, Igor Tufanov, Mias Leontiadis, Fina-Flena Veliche, Itai Gat, Jake Weiss- man, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean- Bann-Btttstee Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthow Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang. Miquel Jubert Hermoso, Mo Metanat, Moham- mad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mchta, Nikolay Pavlovich Laptev, Ning Dong, Nor- man Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zryag- ina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sncha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanic Max, Stephen Chon, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sunit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kochler, Thomas Robinson, Tianhe Li, Tianjum Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Tonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaaiz, Will Constable, Xiaocheng Tang, Xisojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The Llama Herd of Models, 2024. URL https:// /tatpi://arxv..... org/ abtpbb///22. /tbbb//277.278. 21783.
Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabicn Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, Akbir Khan, Julian Michael, Sören Mindermann, Ethan Perez, Linda Petrini, Jonathan Uesato, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, and Evan Hubinger. Alignment faking in large language models, 2024. URL https: //arxiv. org/ abs/2412. 14093.
Melody Y. Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, Hyung Won Chung, Sam Toyer, Johannes Heidecke, Alex Beutel, and Amelia Glaese. Deliberative Alignment: Reasoning Enables Safer Language Models, 2025. URL https:
ptgg/bs///12..... 16339.
Wes Gurnce and Max Tegmark. Language Models Represent Space and Time. In The Twelfth International Conference on Learning Represendations, 2024. URL http://ppaarrreeeeeeeeeooorrr/ee
in.
Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. Cooperative Inverse Reinforcement Learning. arXiv e-prints, art. arXiv:1606.031:: June 2016. doi: 10.48550/ 10.48550/arXiv.16666 http/aarxiv.....01...3333
Peter Hase, Mohit Bansal, Peter Clark, and Sarah Wiegreffe. The Unreasonable Effectiveness of Easy Training Data for Hard Tasks, 2024. URL https: http://axvvvooooaaa/a/aaaaaaa httg/ab//4001.0/777
36

---

Published in Transactions on Machine Learning Research (08/2025)
Stephane Hatgis-Kessell W. Bradley Knox, Serena Booth, Scott Nickum, and Peter Stone. Influencing Humans to Conform to Preference Models for RLHF, 2025. URL https:://arxie httpp:::/aaaivooggaab/////////// arggaba//001.../ 06416.
Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scout Garrabrant. Risks from Learned Optimization in Advanced Machine Learning Systems. arXiv e-prints, art. arXiv:1906.01820 June 2019.
doi: 1.48550/arXD0 Ma.Xiv.1966.00 1906.01880..
Geoffrey Trving, Paul Christiano, and Dario Amodei. AT safety via debate, 2018. URL https://arxiv..or
g/abs/1805.008 00899.
Hong Jun Jeon, Smitha Milli, and Anca Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 4415-4426 Curran Associates, Inc., 2020. URL htttss//proceediing htw.:/ceedddnnn...nni.......p.ps www.eee....ppaa...eeee faarrrffllllppppe httppp/pe...haaaaa////o////////a 4a6af-Paper. pdf.
Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially obscrvable stochastic domains. Artificial Intelligence, 101(1):99-134, 1998. ISSN 0004-3702. doi: https:
///bb//wg/0000000000000000000000 pi1/8800437709800022xx
URL https://wwwt http///wwccinccirrrett..t..
Zachary Kenton, Noah Y. Siegel, János Kramér, Jonah Brown-Cohen, Samuel Albanie, Jannis Bulian, Rishabh Agarwal, David Lindner, Yunhao Tang, Noah D. Goodman, and Rohin Shah. On scalable oversight with weak LLMs judging strong LLMs, 2024. URL https://arxiv, wwwg/bs////eeeeeeeeoeeee 0300704662..
Akbir Khan, John Hughos, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefen- stette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez. Debating with More Persuasive LLMs Leads to More Truthful Answers, 2024. URL https hwtp:dv..rrgabb/eeeeeeee
S.M. Lane. Categories for the Working Mathematician. Graduate Texts in Mathematics. Springer, 1998.
ISEN 9780387984032. URL https: http:///book..oogle..ee wwwpoolks/..d-/oooaaaa.....
Leon Lang, Davis Foote, Stuart Russell, Anca Dragan, Erik Jenner, and Scott Emmons. When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback. Tn Advances in Newral Information Processing Systems, 2024.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a rescarch direction, 2018. URL https://arxiv http:////axxv..og/b/////aea (ttaab///111.08/1101
Kevin Liu, Stephen Casper, Dylan Hadnield-Menell, and Jacob Andreas. Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?, 2023. URL https:
//arxiv.org/ http://bb/3312..0000 03729.
Monte MacDiarmid, Timothy Maxwell, Nicholas Schicfer, Jesse Mu, Jared Kaplan, David Duvenaud, Sam Bowman, Alex Tamkin, Ethan Perez, Mrinank Sharma, Carson Denison, and Evan Hubinger. Simple probes can catch sleeper agents, 2024. URL https://www.artbrrppi..cmm http:/ppp....oae///s//rrpreeeses http://wwwwpprbeeeecccsss--eeeee cateeess--ccchssseeppp r-agents.
Alex Mallen and Nora Belrose. Balancing Label Quantity and Quality for Scalable Elicitation, 2024. URL https://arxin wttps//.rrg/bbb/////ee// /2400. 122111121...........11111111 13215.
Henrik Marklund and Benjamin Van Roy. Choice between partial trajectories: Disentangling goals from beliefs, 2024. URL https://arxdvv.rrgg wrtp//////////././/////// .26990.
Samuel Marks and Max Tegmark. The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets, 2024. URL https://arxivvorrgabb//// 06824.
Julian Michael, Salsabila Mahdi, David Rein, Jackson Petty, Julien Dirani, Vishakh Padmakumar, and Samuel R. Bowman. Debate Helps Supervise Unreliable Experts, 2023. URL https://arxil: www.xivooog/ab//////
1.8002.
37

---

Published in Transactions on Machine Learning Research (08/2025)
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic Regularities in Continuous Space Word Representations. In Lucy Vanderwende, Hal Daumé III, and Katrin Kirchhoff (eds.), Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 746-751, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL https://aclanthology htg.:///aaaatooogoooo/o////oo000 org/N13-1090/.
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent Linear Representations in World Models of Self-Supervised Sequence Models, 2023. URL https://arxiv: httpp///arriio.orgaa/////////// wttp///209.0000....
Andrew Y Ng. Stuart Russell, et al. Algorithms for Inverse Reinforcement Learning. In ICML, volume 1, pp. 2, 2000.
OpenAl. Introducing ChatGPT.
2022. Accessed: 2024-02-06.
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad,, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cumminns, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Elcti, Tyna Eloundou, David Farhi, Liam Fodus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Montijo-Lopess Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Tukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Ben- jamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, 1lya Sutskever, Jie Tang, Nikolas Tezak, Madcleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizaboth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wicthoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang,
38

---

Published in Transactions on Machine Learning Research (08/2025)
Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. GPT-4 Technical Report, 2024. URL https://arxiv.orgg wttp:///////...../////33
Long Ouyang, Jefs Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Joon Sung Park, Carolyn Q Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Perey Liang, and Michael S. Bernstein. Generative Agent Simulations of 1,000 People, 2024a. URL https: http://///wxo.o./////baas/e
Kiho Park, Yo Joong Choe, and Victor Veitch. The Linear Representation Hypothesis and the Geometry of http://axxvv..r//bb....a /2311. 11..1..0000 03658.
Large Language Models, 2024b. URL
Peter S Park, Simon Goldstein, Aidan O'Gara, Michael Chen, and Dan Hendrycks. Ai deception: A survey of examples, risks, and potential solutions. Putterns, 5(5), 2024c.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arriv e-prints, 2023.
Fabien Roger. Coup probes: Catching catastrophes with probes trained off-policy. https://www alignmen Wtgg:i/i/ngSSLLykaaaPPPPP wwwppooobspspppeeeeeooooeoe tforum. Accessed: 2023-04-25.
witppphhssswttthhheeeeeeeee -ith-probees-raain
ed-off, 2023.
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators, 2022. URL sttps://arxiv.. www.///////....../////// 2stp//22..00000.....2
Jérémy Scheurer, Mikita Balesni, and Marius Hobbhahn. Technical Report: Large Language Models can Strategically Deceive their Users when Put Under Pressure. arziv e-prints, 2023.
Rohin Shah, Pedro Freire, Neel Alex, Rachel Freedman, Dmitri Krasheninnikov, Lawrence Chan, Michael D Dennis, Pieter Abbeel, Anca Dragan, and Stuart Russell. Benefits of Assistance over Reward Learning, 2021. URL https::/openrree
Yuda Song, Gokul Swamy, Aarti Singh, J. Andrew Bagnell, and Wen Sun. The importance of online data: Understanding preference fine-tuning via coverage. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 12243 12270. Curran Associates, Inc., 2024. URL https: httpss//proceeednng aper_ www.illesppper//22/2 htte//ile//////e// http:////111e////eaeeeeeeeee
courips.cc/f Confereeneo .dff.
Nisan Stiemnon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human foedback, 2022. URL attps://arks
v. v.org/abs/2009 01325.
Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. Easy- to-Hard Generalization: Scalable Alignment Beyond Human Supervision, 2024. URL https://arxiv.or g/abs/2403. 24422.97799
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Authony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subra- manian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Tliaan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez,
39

---

Published in Transactions on Machine Learning Research (08/2025)
Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Chat Models. arziv e-prints, 2023.
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering Language Models With Activation Engineering, 2024. URL https://arxiv.or org/ abs/2308. 10248.
Elise van der Pol, Daniel E. Worrall, Herke van Hoof, Frans A. Oliehoek, and Max Welling, MDP Homo- morphic Networks: Group Symmetrics Reinforcement Learning, 2021. URL https://arxiv.. org/ //b///////// 2006.16908.
Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept Algebra for (Score-Based) Texl-Controlled Generative Models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances Neural Information Processing Systems, volume 36, pp. 35331-35349. Curran Associates, Inc., 2023. URL https://proceed::
nwwrips. http://appee.lleeeepeepeeeeeee http://sssaaappee////////e//e/e 1252146633211444 Tccb58e5496828f Pawpaeeee88---aee---aaee Confereneee pdf.
Francis Rhys Ward, Francesco Belardinelli, Francesca Toni, and Tom Everitt. Honesty Is the Best Policy: Defining and Mitigating AT Deception. arric e-prints, 2023.
Jiaxin Wen, Ruiqi Zhong, Akbir Khan, Ethan Perez, Jacob Steinhardt, Minlie Huang, Samuel R. Bowman, He He, and Shi Feng. Language Models Learn to Mislead Humans via RLHE, 2024. URL https: www.ivv.rrg/bbb/////00 12822.
Marcus Williams, Micah Carroll, Adhyyan Narang, Constantin Weisser, Brendan Murphy, and Anca Dragan. On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback, 2024. URL https: //arxiv. aba//g/bbb/aa htw.a/////.......
Tan Zhi-Xuan, Lance Ying, Vikash Mansinghka, and Joshua B. Tenenbaum. Pragmatic Instruction Following Srnggedd-Guiied and Goal Assistance via Cooperative Language Guided Inverse Planning, 2024. URL https://arxiv.. g/abs/2402. 11200..17933 17930.
Brian D. Zicbart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Dieter Fox and Carla P. Gomes (eds.), AAAI, pp. 1433 1438. AAAI Press, 2008. ISBN 978-1-57735-368-3 URL http://dblpp uni-trier de/db/ conf httt//aaaaaaaaaaa html#Zie bartMBD08.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geolfrey Irving. Fine-Tuning Language Models from Human Preferences, 2020.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation Engineering: A Tow-Down Approach to Al Transparency, 2023. URL attps://arxiv. wrw.:/s//aaabbaaaeee 23100004000.......
Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving Aligument and Robustness with Circuit Breakers, 2024. URL https:/ /ttp:///axiviiiii wwwpxiv.or//bb///00000
40

---

Published in Transactions on Machine Learning Research (08/2025)
Appendices
In the appendices, we present mathematical details and content that goes beyond the main text. Appendix A lists preliminary results on linear algebra together with their proofs. In Appendix B, we present theory of balanced belief models and the ambiguity for feedback that is given by binary choices between observations. This complements the core theory from Sections 2 and 3, where the feedback is given by observation return function. In Appendix C, we complement Section 3 by showing that human belief models and their morphisms form a category, and we construct variety of natural models composing a diagram. Appendix D contains further mathomatical details for the example Section 3.3 on symmetry-invarian belief models and reward functions. Finally, Appendix E interprets some of the related work from Section 4.2 our framework.
A Preliminary results on linear algebra
For general notation and conventions on linear algebra, see Section 2.1.
A.1 Different representations of linear functions
Let X,Y be two sets. Then by Lin(R*,R') we denote the set of all linear maps F. R - R By Maps(Y, R)), we denote the set of all (simple) maps, or functions, f Y - R Intuitively, these encode the same information: A linear function F, when represented as a matrix, is a collection of rows indexed by y Y, which are "picked out" by function f Y Rx This correspondence is made precise in the following proposition, which is "transposcd" version of the classical statement that linear functions vector spaces correspond to functions basis: Proposition A.1. Define the two functions
lin
Maps(Y,R*)
Lin(RX R
map
as follows: For f E Maps(Y,R*), v R and y E Y, we define
[In()] (f(y),))
For F E Lin(RX, R EYY and a E X, we define
(m)) (x) = Fyz.
Then lin(f) has matriz elements
lin(f) [f(y)]
for x E X and y E Y. Furthermore, lin and map are mutually inverse bijections.
Proof. First, note that for each f E Maps(Y, R²), lin(S): R - RY is indeed a linear function since the scalar product is lincar the second component. Its matrix clements are given by
y = [[im([]] (e (v) (f(w), [f(]
Now we show that lin and map are mutually inverse bijections, i.e. lin map id RR) and map lin idMapa(Y.RA)) Indeed, for F E Lin (R),, æ X, and y E Y, we have
[lin (F)) (in(map(F)))
41

---

Published in Transactions on Machine Learning Research (08/2025)
[[in(maa(F))](aa]]
( [[map(F)] (v).
[map(F)) [F) (v) (x)
Fgg
Since linear functions fully characterized by their matrix elements, this shows (lin (F)) F, and thus lin map is the identily.
For the other direction, for f e Maps(Y, y E Y, and E X, we have
[[mmao lin) [[map((nn([]]
lin(f)
(x).
This shows that (map o lin = f, and so map o lin also the identity.
Proposition A.2. Let X, X,Y Y be sets and F RX + RY F: R +RY and & R + R be linear functions. Let f = map(F)) :Y Y + R and j= map(F) : Y - R be the functions corresponding to F and
F by Proposition A.1. Let x² R + Rx be the transpose of 4, with matric elements F Fo00 &b if and only if f = th f:
x² a Then xx
R %
R x
4
7
R
R Y
R X
Y
F
f
Proof. We have
FoFoo +
YyEY, Fyg 1) Σ
F-d=
y
2EX
YyEY, [z)]) = Σ Φ2 (()) = ( f())
TEX
x0 (fw)) (x)
[10" (w))
f= f= of.f.
That show.
A.2 Properties of kernels and images of linear functions
The following two propositions list basic and well-known properties of kernels and images that we use in the paper:
Proposition A.3. Let A V + U and B V + W be linear functions. Then the following statements are equivalent:
1. ker(A) (A) C kee((B))
42

---

Published in Transactions on Machine Learning Research (08/2025)
2. There crists lincar function C u - W with C' A B:
B
W
A
U
Proof. Clearly, the second claim implics the first. So now assume 1. Let fu!...., = be a basis for im(A) and complement it to a basis {u',..., )²) for all of U, where n 2 m. For each i E (l...., m.), let o² be
any element with A(v') u'. Define ((²)) B(v') for i {l....,m, m} and C(u') if m. Linearly extend to linear function C:U-- W.
To show that Co A = B, let V be arbitrary. Then A(v) im)(A, and thus there exist coefficients A R for E (1....,m) m) with
A(v) Σ
Au'
1=1
Note that
A
A(v) Σ A(((')== A(v) - Σ
Aau = 0
1=1
1=1
Thus, v- -m dss E ker(A) S (B) Consequently, we obtain
m
m
B(v) B
Σ
Σ A₂ Σ A.((m))
m
=C
Apu
C(A()) (C A)(o).
This shows
B, and thus the claim.
Proposition A.4. Let A:4 + W and B: W be linear functions. Then the following statements are equivalent:
1. im(A) C iB).
2. There erists "lift", i.c., linear map C:U - V with BoC= A:
V
W.
Proof. It can casily be chocked that the second claim implics the first. For the other direction, let ful .u....., un} be a basis for U. For E (I,...,n], choose o¹ E V with B(u") = A(w'), which exists since im(A) C (B). Define C. :: + V as the unique linear function with C(u') o¹ for i e nI. We obtain
A(u') = B = (B (u²).
Since linear functions determined basis, it follows A Bo C, proving the claim.
43

---

Published in Transactions on Machine Learning Research (08/2025)
B Balanced human belief models and choices
In this appendix, we present the core theory from Section 2 and Section 3 for the case that the feedback is in the form of choices instead of the observation return function Go. To still get a useful theory, we will then need to assume our human belief models to be balanced to ensure that constants are "propagated" appropriately through the model. In this whole appendix, we fix an MDP with set of trajectories and observations O.
B.1 Balanced belief models
Definition B.1 (Row-constant). Let X, and Y be sets. We call a lincor function A: R RY row- constant if for all y. E we have 0 + Exex Xy== = Xya- Definition B.2 (Balanced). Let M (Ω, A, V) be human belief model. We call M balanced and & are row-constant, and if R contains all constant functions.
A and & being row-constant means that the corresponding functions X: RD and O - R (cf. Propo- sition A.1) map to vectors of feature strengths with constant total weighting, as is for example the case for probability distributions. That V contains all constant functions is often naturally the case. To demonstrate this a simple example we first prove the following lemma:
Lemma B.3. Let X, Y, be sets and A: R + RY, B: RY + +R² be row-constant linear functions. Then the composition B A: Rx + R is also row-constant.
Proof. Let a, be the row-sums of and B, respectively. Then for all Z, we obtain
AB) EE
Bzy
BBy
Σ Agz = b.a b
#EX
REX WEY
VEY
ZEX
which shows the claim.
Example B.4. We continue Ezample 2.6 and show that the model is balanced. Note that for all & E E, we have
Σ
I T((()1)))))
Σ
Σ
( S141)
(SSx,")
(Exx,") t=0
Σ
"1 aes 8141)
(xxxAx)
# 0.
Thus, I is row-constant. B is also row-constant since all rows are probability distributions, and so Lemma B.3 implies that also & = Bor is row- con-constant. V = AxS also clearly contains all constant functions. Overall, this means the model (S x BoT, SxAXS) is bolanced.
For more examples of balanced human belief models, see Appendix C.2.
B.2 The ambiguity for balanced belief models and choices
Let σ : R + (0, 1) be any known bijective function with o(r) + (o(-)) = 1 for all r E R. (e.g., the sigmoid function). For o, o' O, we define the probability that a human with feature belief function E: R R and reward object Rq Rº prefers over of by
pña - 0') = -o ([)(()))( [[((o)]))))
(20)
44

---

Published in Transactions on Machine Learning Research (08/2025)
For the rest of the section, we fix human belief model M (2, A, Furthermore, we fix the implicit true reward object Rn E V together with the return function G = (Ra), the observation return function Go = (()) and the choice probability function Po = Pe that serves as our operationalization of "foedback".
We use the following adaptation of Definition 2.8:
Definition B.5. We define the set of return functions that are feedbackk-compttibue with Po as
(PM)(P)) = G 3Za E V: Pe = Po and A A(Ra) G
We define the ambiguity lest in the return function G after the choice probability function Po is known by
Amb (G, (G, Po) = G' G' = = G for & (PM))
Clearly, we have
FCM((G)) = G Amb Po).
Recall the ambiguity Amb Go) defined in Definition 288. We obtain:
Proposition B.6. Assume M A,,,,, is balanced. Let 1 E R= denole the function that is constand
1. Then:
AmbMiG,Hn = Amb Go) (o06)))+(((1/ee (c.I/CER) A(ker(E) + (oe))+((ec/))
Proof. The second equality follows from Proposition 2.9, so we are left with proving the first. By abuse of notation, will write for the three functions that constant 1 E, O, or f.
Let G' E (G, Then G = A(Ra) for Ro E V with pfa- = Po. The latter means the following for all o e EO:
o ( (o(Ra)(oo) o ([E(HR))) [(()))](6').
Since is invertible, obtain
( [8( [8(Re) (o')
Fix any o' E O and set co = [E(Ro)] [E(Ra)] (o'). Then for all E O, we have
[E(Ra)] = [E(Ra) +
Or, equivalently:
E(R)) = E(Ro) + co = Go + co. 1. Since & is row-constant, there exists en E R with E(co 1) = co 1. This implies
E(Ra co 1) Go.
We also have Ra on 1 E since V contains all constant functions. Furthermore, A(ca 1) = c 1 for another constant C= IR since A is row-constant. Overall, thus obtain
G¹ = A(Ra) - - en 1) - 1E Amb AG) (c-
For the other direction, let G" E Amb (G, + fc c CEER. Then G' G for G = A(îa) with Ro E V with E(ft) Go. We have
G 1= A(Ra en cn.
45

---

Published in Transactions on Machine Learning Research (08/2025)
for constant ca E IR with A(co cg -1 Since V contains all constant functions, we have Ratep 1 E V.
We also have
pe Plateut = Pe Po
since the constant gels cancelled out in the definition of the choice probabilities, and since E (RR)) = Go- All of this implies
G' = +G G Amb Po). Po).
That proves the claim.
Note that for the purpose of policy optimization it is not an issue that the ambiguity has an "irreducible" constant term since this does not change the ordering policies under the policy evaluation function J(T) = (EEP=()|G(E)
Remark B.7. In light of the previous proposition, it turns out that the ambiguity does not depend on the true relurn function and choice probahililies, and we con thus write it as Ambp = Amb (G, Po). The "p" is added to distinguish from the ambiguity Amb Amb (G, Go) that we study in the main poper.
Using this result, we also obtain version of Theorem 3.2, with the ambiguity replacod by the one we use this appendix:
Theorem B.8. Let. M (22 E,,,), and M = (Q, be lwo balanced human belief models and assume that M covers M. We think of M the "truc" human belief model with reward object Ra E V and corresponding relurr function G = A(Ra) and choice probability function Po = Pel.. Then we have:
Ambş Ambp.
2. If M also covers M, then Amb Ambp.
3. There is an R E D with P R
Po and (R =.
E
Rr 4. Assume M is complete. Then every reward object R2 E V with P G for constant ez R.
= Po also salisfies A(Rag) =
Proof. Statements and 2 follow from the ambiguity characterization in Proposition B.6 and the analogous statements in Theorem 3.2. Statements 3 and 4 can be proved with similar arguments the corresponding statements in Theorem 3.2.
C A diagram in the category of human belief models
Let us consider an MDP together with a fixed set of trajectories E and observations O. Then in Definition 2.4, defined the notion of human belief model M (2,, A, E, V). In Definition 3.3, we then introduced the notion a morphism
M + M between human belief models, which is defined as a linear function ① R + R such that &(V) V 4 A and E 4 E. This notion turned out important since it is oquivalent to model covering (Definition 3.1), which implics that the covering model can be used for the return function inference from human feedback (Theorem 3.2), especially if its ambiguity disappears.
Belief models for fixed sets of trajectories and observations O, together with their morphisms, form a category (Lane, 1998), meaning that they satisfy the following simple properties:
Composition: Assume MI, M3 are three human belicf models and T MM + M2, o' M2 + M3 morphisms between them. Then also the composition o/o : Mi - M3 is morphism.
Identities: For any human belief model M (9,A,E,)) V). the identity idga M + M is a morphism.
46

---

Published in Transactions on Machine Learning Research (08/2025)
Associativity: (&" (") ob o ( Φ) for any three morphisms that can be composed in the specified order.
All of these properties can be trivially checked, and so human belief models and their morphisms indeed form category.
In this appendix, we want to write down simple commutative diagram of morphisms in this category. Here, diagram means a graph of human belief models and morphisms between them. For this to be commutativv means that any pathway from one human belief model to another is the same morphism. We prepare this Appendix C.I by writing down all linear functions from which the functions A, &, and + will be constructed. In Appendix C.2 we then specify the resulting human belief models and briefly consider their properties. In Appendix C.3 we interpret the matrix elements that appear in the feature belief functions of all models. Finally, in Appendix C.4, we write down the resulting commutative diagram and the resulting relations for the ambignities.
C.1 Preparing the models
We build on Example 2.6. The idea is that we consider reward objects at four different. levels: Return functions, classical reward functions, and return- and reward functions of abstractions of trajectories and transitions that the human might care about. By modeling the human as having features at all four of these different levels, we can create a multitude of human belief models.
Let B I= Rº be the matrix corresponding to trajectory-belief function b O + A C R via Proposition A.1. Let r RXXAXS + R be the linear function mapping reward functions to their corresponding return functions.
Let FF be a set of "abstractions of transitions" and h: x A x S + F a function mapping each transition to its abstraction. Write reward objects over abstractions as Ry E RR.. Then we obtain the induced map
h" RR SXAXS, R Ry o
h (Rx) measures transition ss)),s') by evaluating Ry at the transition's abstraction: Ry(h(s, a, Thus, he' (Rx) is guaranteed to give the same reward to transitions with the same abstraction.
We can then also consider the space of abstraction sequences Jf together with the function H² E + F given by
h" (so: ao..... ST- -T-1 ST) = (h(s0, ao, s1), .., ,. (1ST-1,0T-1,5
Write return functions over abstraction sequences as GJT RF ht then gives rise the dual function
F" + R GxT +
Thus, hTT (Gft) evaluates trajectory by evaluating the sequence of abstractions using Gyz, As before, two trajectories with the same sequences of abstractions then obtain the same return.
Recall the function r SxAxS + R mapping a reward function to the corresponding relurn function. Then we obtain an analogous function for reward objects on abstractions:
T-1
R RF
[F
fT) Σ (f))
Proposition C.1. The diagram
SSXAS
R
R
(21)
M*
RF
R
T3
of linear functions commutes, meaning that all pathsways with the same slart and end are the same function.
47

---

Published in Transactions on Machine Learning Research (08/2025)
Proof. We have
(T oh (Ry) (5p, du....,.........................
r [hh (By))) au........... au ar...,,
Σ 2' 2Ih² (s,,
Σ n' Ry (h(s,,
t=0
[>(())] (1""))))))))) (80, 1...00,......................... 11
hnT (Tx (s0. ao ................................ .........011111111 -1
[(h"" o [F) (RF) ao....,
.......,T--. -1, ST).
This shows P'h" hT Fr. Consequently, the diagram commutes.
The idea will be that the rows of B, BoT, Boh" and Bor = B oh of' all correspond (via Proposi- tion A.1) to feature beliefs over trajectories, transitions, trajectory abstractions, and transition abstractions, respectively. We explain this interpretation detail in Appendix C.3. All these functions "factorize" over trajectories, but course this need not be the case reality: A realistic human could have an intrinsic belief over state transitions, sequences of abstractions, single abstractions, without this belief "factorizing" in rational way over state sequences.
Thus, let the following be an extended version of the diagram from Proposition C.1, with new linear functions B', B", B" This extension is now not necessarily commutative anymore:
B
SXAXS
r
R
B
R
(22)
h
B
R
FF
R
F
8"
To interpret B', B", "" similar grounds B, it makes sense to assumne that they are row-constant
(Definition B.1), but otherwise they can be arbitrary.
C.2 Various human belief models
We take the previous diagrams the starting point to construct human belief models Remember that a belief model is of the form M (S2 A, &, V). The sets S, x A x S, F, and FI are four different possible feature sets Ω. A is given by a composition of linear functions that maps to R=. E is given by composition mapping to R The space V is either given by the full vector space R or by images of functions mapping to R Overall, using the diagram from Proposition C.I, this leads to the following 9 models, with the superscript denoting the feature space, and the subscript indicating where the valid roward objects "originate from":
MF = (F, Poh", BoTch" R MxxAx (8 A r, Bor, RSXAXS)
48

---

Published in Transactions on Machine Learning Research (08/2025)
M (S A x S, r, Bor, im(h"))
M5T (FT, ht, Boh"" R²)
M = (FT, n1",, Boh", im(T>))
M [E, idRE, B, R²
M5 AxS (=. idRa B, im(T))
MaT (E. idR=, B, im(h"")
MI (E, idRe, B, im
For example, S is the model from Example 2.6; M M5XAxS the same model, but with valid reward functions restricted to those that only "care about" abstractions; MI is a model in which the features are given by full trajectories, and there are no restrictions on the valid return functions; etc.
Now, B' naturally gives rise to the following three models, which color differently distinguish them more easily:
MT = (F, Toh', B'oh", R²) M = (8 A x S, P B', im((i))
MISXXS SxAXS (S A 8, r, B', RSXAxS).
Similarly, B" gives rise to the following three models:
MT = (F, 1T o Tr, B" Tr, R))
M (FT, 1T, B", im([z))
M (FT, 1Tt _B, R²)
Finally, "" gives risc to a single model:
M (F, Toh", B"", R²)
Note that all component linear functions appearing in any of these models (identities, h", T,Tr, AT" B...., ......, are row-constant. By Lemma B.3 then, also all compositions are row- constant, which then implies that all 16 models are balanced, as defined in Definition B.2. The first 9 models are also faithful (Definition 2.15) since all feature belief functions factorize as in Proposition 2.16, with Y given by B in all cases. The other 7 models will typically not be faithful.
C.3 Interpreting the matrix elements
We now interpret the different feature belief functions that appeared in the nine first models of the previous subsection. Recall that the linear function B R= - R "comes from" function b O + A(E) C R Thus, all matrix elements Bog can be interpreted as probability [b(o) (E) for the trajectory & when viewing observation o. We now explain similar interpretations for the matrix elements of all the other feature belief functions:
Bo P: It contains matrix elements
(B BeDDo
Σ
[E)
Σ 2 ( at, 81+1),
t=
the expected discounted number of times the transition (s, s') is present in the trajectory.
49

---

Published in Transactions on Machine Learning Research (08/2025)
B ohh": Write f for (f1..... fr). Then this contains matrix elements
(Boh T )
(S)AE
EE
Σ EEE
[b(0)] (E)ōr(h"
Σ
[b))](E)
E: (((
(f).
In the second to last step, we view b(o) as probability distribution that, when evaluated on a set, evaluates to the sum the probabilities of the set's elements. the last step, we use the definition of the distributional law of a random variable X with respect to a probability distribution P on the sample space: Px(x) = Px)-)(x)). The result is the believed probability, after observing o, of a trajectory with soquence of abstractions f.
Finally, look at the matrix Bo PoToh" (for which give two slightly different formulas): The matrix elements are given as
B (Bo
Σ
(B D Dolnasy Mos00'
(sxx,")
T-1
Σ
(E))(( Σ
rd (' (s,, 81+1)
(s,a,)): M(a,s)) $EEE
T-1
Σ (E)
Σ
(11 as, 8141)
EEE
(8,0,8)): b(s,a,)))
T-1
Σ EEE
[E)
Σ
(6
This is the expected discounted number of times that one encounters the abstraction f. Using that Poh" T o or by Proposition C.1, we can also write this as
(Boh" oFF), = Σ (Bol" Jor (TF)es
PEFT
[(O) (f)
(f) Σ r (fi).
FEFT
t=0
This can also be described as the expected discounted number of times that one encounters the raction
f.
C.4 The resulting commutative diagram
Building on Appendix C.2, the following is a commutative diagram of belief models and model morphisms, with four differently colored "connected components"; one can sometimes use Proposition C.1 in the process of showing that every linear function in the diagram is morphism of belief models, and that the final diagram commutes:
50

---

Published in Transactions on Machine Learning Research (08/2025)
MISXAX SXAXS
M MSXAxS SxAX8
M SxAXS
id
M
idaxxXx
id
kdga
M
M
M
id
M Fr
F
h*
R²'
id RFT
M
M
Fr
M
M
M
M
Dr
My
id nr
MFF Fr
For example, the following diagram visualizes the fact that h² MY
M is morphism:
IR=
Poh"
r
R
h"
R
Ul
im(h")
B.T
Boroh
RR
This gives rise to the following diagram of ambiguitics:
[(ker(B'))
(k)r(BooT)
ker(B) Ber(B)nnn (Im
S
ker(B)
Ul
Ul
r r (oou(B")iim(
P ker o) nim(h"))
ker (B) n
ker B n im (A²")
II
(Tolt"((ur(B) (krr(B'
(T (kor"
AT (ker(B of ) (nn))())))
NTT (ker(B o/i
(kerB)())
(AT FooTF (ker(B"
A (""(ker(B") n (T))))
17" (ker(B"))
The ambiguities computed using Proposition 2.9, and the inclusions and equalities of ambiguities follow from Theorem 3.2 and Theorem 3.5. Here, the ambiguity I (ker(B [B) = ker(B) n im(T) is the special discussed in depth in Lang et al. (2024). Note that the models M5, M MSXAxX and MSxAAx SxAx8
closely
related to the models M1 and My from Section 3.3.
Assume we would use one of these models in practice. The further right or up it is in the diagram, the more ambiguity there is, but it is then also more likely that the model covers the true belief model (should it appear in the diagram in the first place). Thus, there a trade-off between covering the true belief model, and keeping the ambiguity small.
D Details on the example with invariant features
Here, we present more mathematical details for Section 3.3. This appendix is self-contained and we recommend reading it alongside the section in the main paper.
51

---

Published in Transactions on Machine Learning Research (08/2025)
D.1 Details the MDP and observations
Formally, the states are given by S = (LL, x {U, D))?, with the first component being the hand-position, and the second component being the button position. For example, the state in Equation (12) is given by (R, D)).
Furthermore, we define functions POsh + (L, R] x (U, D) and Posp : : [I,, R) x (U, as the first and second projection. These are the position of "hand" H and a "button" B, in 2x2 gridworld. Then the state s from Equation (12) satisfies PosH (s) UL and Posn(s) (R, D).
The set of trajectories is formally given by =\t = (S A)3 x S. The set of observations is formally given by
o= [L. [P.P) R)2
Here, P means that it was not observed that a button was pressed.
D.2 Details on the belief models and symmetries
Let G= = D4 be the dihedral group of order 8, i.e., the symmetry group of the square. It is given by
G= D4 = {e, r, 12, 33 S, rs. SS A
where is clockwise rotations by 90 and I is flip over the horizontal axis. In compositions, apply f first. G' acts S x A by individually acting states and actions:
g.(s, a) = (g.s,g.a),
where g.s = (s), = (g. (os (5), g. Pos where on the generators g = r and g = f we have
rL(LU)) = (R,U), rR(RUU)- (R, D), rR (L, =(L,D)=(LL) = - ((R,U)-
D), ((R, = (R,U), =L,)),
This specifies the action on states. On actions, we specify
rL= U, rU= R, r. = D, rD= = L, TP=
=. fUU f-R R. f.D U, f.P
Thus, the "pressing" action remains invariant.
With this group action, we obtain set of equivalence classes of state-action pairs, given by S x A. A set of representatives for the equivalence classes is given by
(+oo) x ) u [[ A U ((82) x A*²),
(23)
where
80 DR, D)), 81 81 ((L, (R. D)), 82 ((I,U). D)), and where the (state-dependent) set of actions are given by
A00 2 = (L, = = (L,.UDD.
We then have a function
h:SxA
{s)} x * sE(0,1.2)
U
that maps each state-action pair to a representative, given by h(s, a) g-. a) for the unique g E D4 for which (s, a) in the of representatives from Equation (23). Via h, we now identify S x A with (1e (8i) x _
52

---

Published in Transactions on Machine Learning Research (08/2025)
Equation (15) can be showed by
(, = [s())
Σ
6 ae))
0=
E
8 (ho))
Σ (()
(24)
Σ
Mi ((
(x,)))SSAA
E
(e) Mi ())
(s),a))SXA
(Toh") ),,(s.))
(Toh")
For the matrix elements of ht, we used Equation (1)-
The human's belief E(s, a o) for (s.a) S x A and o O is then given as follows: The human has a uniform prior B(s) Po over possible start-states sampled from Po. and uniform prior over possibly next actions given the current state, leading a prior distribution over B(E) E A(3). Then, upon seeing o, the human implicitly computes posterior belief over trajectorics compatible with the observation, simply given by
B(E o) x 6s(O(E)) B())).
(25)
Equation (16) can be showed by
Ev.(..a) [E(O]] as
[b(o) (5) [A(E) (s,
EEE
Σ Bug (g,(),) EEE
(26)
-BB oA)o,
=(B I oh (o,(s,))
D.3
Details on the ambiguity analysis for .M2
In all these computations, recall Equation (6) and the sentence following it for computing (R) for an observation o.
Recall the observation 02:
P
B P
0 = H
B
B
H
B
Since we assumed that the starting state is one of the states in Equation (13), the human has the belief [82, P) = 3, iee, the human is certain that, to symmetry, the hand performed a pressing action
three times in 82. Thus,
0= (02) [c(oz)] (82, P) R(s2, P) = R(s2, P)
This implies R(52, P) 0. Recall observation 01
P
P
=
H
B
H
B
H
B
H
B
53

---

Published in Transactions on Machine Learning Research (08/2025)
Now, the first action could either not change anything, horizontally align H and B. An action that does not change anything is more likely (chance 2/3 since there are two actions, in the direction of two different adjacent walls, that achieve this, which both correspond to action L up to symmetry), and so we obtain
[e(or)] (s2, L) 2/3,
[e(o1)] (s2, P) = 4/3, (s2, - 1/3, [c(or)] (81, P) = 2/3.
Compare also with (26). Thus, we obtain
0 [E(R)] (01)
2//3 R(s2, + 4/3 R(82, P) + 1/3 R(82, D) 2/3 R(s1, P)
2/3. R(81, P).
Here, we used that R(s2, P) 0 by what we showed before, and R(s2. L) = R(s2, D) 0 since R(s, a) =
whenever a + (i.e., since R E v). Thus, we have R(s1, P) = 0 as well.
Finally, we look at the observation sequence on given as follows:
P
=
H
B
H
B
HB
HB
Again, there is a chance of 2/3 that the first action does not change anything. Given the first step, everything which follows deterministic, leading these feature beliefs:
[c(ou)] (s2, L) 2/3,
[c(ou)] (s2, R) 2/3,
[c(ou)] (81, P) 2/3
[e(ou)] (s2, D) = 1/3,
[e(on)] (s1, R) = 1/3, [e(or)] (so. P) 1/3.
Compare again with (26). This means that
= [E(R) (oo)
2/3 1R(2, L) + 2/3 R(s2, Rt) +/3 P)+
+ +1/3- R.(s, D) 1/3 R(s, R) + 1/3 R.,,( P)
1/3 R(sg. P).
Here, we used that R(s1. P) by what we showed before, together, again, with the fact that R(s, a) = 0 for all a + P. That shows R(su, P) = O.
D.4
Details on the ambiguity analysis for Ms
We have
(BoT) = (Bo 11((11,)) () (BoT). ( - R' P)
oBo ()), (Bo ().C..))
(27)
In the computation, the second step follows from the definition of R'. The last step follows from the symmetry remarked before.
E Mathematical interpretations of related work in our framework
In this appendix, we briefly interpret some of the related work from Section 4.2 in our framework for the special case that they learn linear reward probes. Note that these interpretations not meant to
54

---

Published in Transactions on Machine Learning Research (08/2025)
capture everything there is to say about that work the summaries provide are quite coarse. In all examples below, wC assume access to a very capable foundation model x: E + R that allows for a lincar ontology translation V R + R the human's ontology X, as in Section 3.4.1: Y - = A. Define @ = v R + RR, which then satisfies A o + A by Proposition A.2. In all approaches below, we define and assume that the return function is learned with the same method as in Section 3.4.2. Notably, in all of the approaches one essentially just defines ? X, i.c,, no explicit modeling of humans performed.
E.1 Amplified oversight and eliciting latent knowledge
In amplified oversight, one amplifies the human to give accurate feedback, which means we can assume O = and c = X. One approach to achieve this would to essentially define = v o Ā by giving the human access the linear ontology translation for understanding the foundation model's thoughts. This would roughly be in the spirit of cliciting latent knowledge (Christiano et al., 2021), where the human can query reporter to give information about arbitrary latent knowledge of an AI.
Accordingly, also choose ē = X, leading to the following coverage diagram:
R
RRa
R
R
The ontologics and feature belief functions are then the same, which automatically means that the ambiguity disappears: A (ker(A)) 0.
E.2 Easy-to-hard generalization
In this setting, O O E is subset of trajectories that the human correctly understands. Thus, for & O, one has <(g) = X()), and so Alo is simply restriction. In this setting, one can also set ē = Xlo. Now, let A| and A| be the linear functions corresponding to Alo and Nlo, respectively, via Proposition A.1. One obtains the following diagram:
R
R
R
Ä
A
RO
For & in this diagram be morphism, we need that the lower diagram commutes. With Proposition A.2, this follows from the assumption that क. T is ontology translation: V VoA= = implics v o Xlo Alo- The ambiguity now given by Â ker (A)) By using Theorem 2.13, this ambiguity disappears if and only if for all & E E, we have Ā(g) E { EçEz ZęX(E) Zg R]. Thus, the ambiguity vanishes the trajectories that the human understands have enough variety in the vector space of feature strengths.
E.3 Classical RLHF and teak-to-stront to-strong generalization
In classical RLHF, without any safeguards, one just uses the model é À as the feature belief function even though * À and hopes for the best:
55

---

Published in Transactions on Machine Learning Research (08/2025)
R=
R
1
R
(28)
R
In this case, the lower triangle does not commute since A t + E, which, using Proposition A.2, is due V Voλ= x e. This means that the second model does not cover the true belief model, and the guarantee from Theorem 3.2 breaks. In fact, the return function that would be inferred using this model is Go, the observation return function itself (ef. the definition of feedback-compatibb return functions, Definition 2.8, applied to this faulty model). Lang et al. (2024) extensively discuss failure modes in this case, called deceptive inflation and overjustification.. We note that weak-to-strong generalization (Burns et al., 2023a), when used without additional techniques, also considers this setting, but trics to ensure that the learning process or X contains inductive biases that steer the learning process to learn G anyway.
56


Reviews:
[{"review_id": "vLzXgaCRi9", "reviewer": "Reviewer_MZRf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Weaknesses**\n\nWriting and formalism: The paper is difficult to read because the mathematical formulas are defined once and not referenced with simple reminders in the text. For example “Even if the human could fully compute λ(ξ)…” would be more reader friendly if it was “Even if the human could fully compute the ontology of the trajectory ξ, λ(ξ)…”. Also, the algebraic and category theory terminology is inaccessible for the vast ML community, but I appreciate the effort the authors put in making these concepts as accessible as possible.\n\nThe Speculative Nature of a Belief-Simulating Model: The practical proposal in Section 3.4 hinges on the ability to construct a feature belief function that accurately simulates the human's beliefs and blind spots. The paper suggests methods like \"belief prompting\" or using earlier model checkpoints, but acknowledges this is highly speculative. As the authors note, it is unclear whether such techniques can reliably shape a model's internal representations to match a human's cognitive limitations, especially when other work shows models can have latent knowledge of the truth even when outputting falsehoods. This is the most significant leap of faith required for the proposal to work.\n\nLienear assumption: The entire framework, including the decomposition of rewards and the use of linear maps, assumes that human preferences are linear in a given feature space. The authors acknowledge this as a potential area for future work, but it is a significant limitation of the current theory. Many complex human values are likely non-linear, and it would strengthen the paper to discuss the potential implications of this assumption in more detail in the main text.\n\n**Strengths**\n\nThe framework is appealing and mathematical rigorous: Besides the mathematical heaviness though, the paper is very well written with a lot of educational content to give context and bring the reader up to speed with the terminology. The introduction of \"human belief models\", the formal characterization of \"ambiguity\" in terms of linear algebra, and the precise conditions for its disappearance (\"completeness\")  are well-defined and well-argued. This formalism allows for a clear and unambiguous analysis of the problem.\n\nThe paper frames the scalable oversight problem not just as one of partial observability, but as one of flawed human belief. This is a crucial and insightful distinction. By proposing to model the discrepancy between an AI's actual behavior and what a human believes its behavior to be, the work provides a new and powerful lens through which to view and address potential AI deception and misalignment arising from human limitations.\n\nPragmatic Relaxation of Assumptions: A key strength is the move from the unrealistic assumption of a \"known true model\" to the more plausible goal of finding a \"complete covering model\". The introduction of belief model covering, morphisms, and ontology translations provides a sophisticated theoretical toolkit for reasoning about this more realistic scenario. This demonstrates a thoughtful approach to bridging the gap between pure theory and potential application.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a framework that models error in perception or belief in RLHF. The human is assumed to give feedback that is perfectly rational given their flawed understanding. This is known as scalable oversight in the literature.\n\nTo achieve this, this framework proposes turning trajectories into features and then using beliefs and rewards over those to capture the preferences of humans. However there are cases that capturing the preference of the human is problematic:\n\nAmbiguity: when the value system (reward function) of the human is recoverable / identifiable from the feedback because many different reward functions could have produced the same feedback. The paper suggests that the solution is to have a set of observations for which the collection of human beliefs is rich enough to cover all the different true feature vectors that can be produced by the ontology.\n\nFaithfulness: if any two value systems that are indistinguishable in reality are also indistinguishable in the human's feedback.\n\nThe authors define a \"complete\" belief model as one for which the ambiguity disappears. They provide a clear sufficient condition for completeness: a model is complete if the feature vector of any possible trajectory can be expressed as a linear combination of the belief vectors from the set of observations. This provides a clear target for designing practical feedback-gathering processes.\n\nRecognizing that the true human belief model is unknowable, the paper introduces the crucial concept of \"human belief model covering\". The core idea is to specify a known belief model that is expressive enough to represent all possible return functions and feedback that could be generated by the true, unknown model. The authors show that if this covering model is also \"complete,\" it enables the correct inference of the human's values from their feedback, providing a practical path forward for the paper's theoretical framework.\n\nThe paper bridges the gap between deep theory and practice by proposing a method to construct a covering model using foundation models. The novel idea is to use a powerful, state-of-the-art model to represent the ontology and a separate, belief-simulating model (e.g., an earlier checkpoint or a prompted version) to represent the feature belief function. This proposal, while speculative, provides a possible direction for future empirical research.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Clarify the Speculative Nature of the Practical Proposal**: The theoretical contributions of this paper are its strongest aspect. The proposal in Section 3.4 for instantiating the framework with foundation models is compelling but highly speculative, relying on the challenging task of creating a \"belief-simulating\" model that accurately mirrors human cognitive limitations. I would recommend the authors more clearly delineate the established theoretical results from this more forward-looking, hypothetical proposal. Adding a dedicated \"Limitations\" subsection to discuss the significant empirical hurdles in creating such a belief-simulating model would help manage reader expectations and highlight the most important avenues for future work.\n\n**Expand on the Linearity Assumption in the Main Text**: The authors correctly note in the future work section that their framework assumes the human's return function is linear in the features. Given how central this assumption is, I believe the paper would benefit from a more prominent discussion of its implications in the main text. For example, many crucial human values (e.g., regarding fairness, safety, or avoiding catastrophic outcomes) might not be simple linear combinations of features. Briefly exploring why this assumption might be a significant limitation would provide important context for the scope of the current framework's applicability. Perhaps a suggestion - could a non-linear theory be exposed via the use of information theory (e.g. mutual information between ontology and beliefs)?\n\nNit: This paper reminds me a lot of POMDP-IRL however I understand that there are some differences (e.g. assumptions about optimality or deceptive nature of AI systems). I’d like to some some thoughts on that.", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DG4MmHJFTo", "reviewer": "Reviewer_25yK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the paper is well-written. It might not be easy to follow for people who don't have strong math background or who don't read it carefully, but I think the authors handled all the notations, setups, and theorems elegantly, and I appreciate the authors' efforts in this. \n\nI don't have much question or requested change for the paper. But I'd like to raise a few points that arise when I read the paper. I hope the authors can think about these questions. \n\nThe paper is really mathematical. We are not short of empirical papers at all. The LLM/GenAI community does need mathematical papers to clarify the foundation of our current models and methods. But what would be the audience of the paper? \n\n- For ML people, this paper is too technical for them to draw insights from, or I think the ML engineers working on the problem hardly understand the paper. For me, I want to be honest that given I think I am relatively on the theoretical side of the ML community, I tried hard and understood the contents of Section 2 but got lost in Section 3.2; the example of Section 3.3 is good. So if the authors want ML people to read and understand the paper (though you don't have to if you don't want), you need to write it in more plain language. Also, the MDP formulation of human feedback/RLHF is reasonable but over-complicates the understanding for the general ML audience. You don't have to rewrite the paper; just make one extra section in the very beginning, summarizing the key insights and take-away messages in layman's terms. And that would be great. \n\n- I think the paper is a good example to attract more mathematicians or people with math background to work on LLM/GenAI-related problems. Just like this paper, I don't think we have a satisfactory mathematical definition of \"hallucination\". I'd love to see a paper on that topic, just like this one.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a comprehensive and novel mathematical framework for preference inference/learning through human feedback. It considers an MDP setup for the reward modeling and introduces human belief model to formalize the human's \"annotation\" of the data. The paper mainly addresses the question of under what conditions, we can infer the reward function via human feedback, and proposes a full rigorous framework to characterize the mathematics behind the problem. The discussion comes with examples to better generate intuitions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.", "broader_impact_concerns": "Nan.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AKSVD7ojKZ", "reviewer": "Reviewer_YrGE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "To be honest, I feel it is very difficult to assess this paper. I guess the reason would be either 1) this paper is really innovative or 2) this paper is not well-written and hard to understand. But, let me summarize the Strengths and Weaknesses.\n\n### Strength\n- This paper seems to pose some novel ideas for discussing scalable oversights. Since AI safety has recently been a very important topics in our society, this paper can be a good anchor for the future research and development.\n- If I understand correctly, the theoretical analyses looks novel. The proofs are well-written (I didn't check all the proofs, though). \n\n### Weakness\n- Introduction is well-written and fun to read, but there is a large distance between what is discussed in the introduction and the essential contributions in this paper. I am wondering whether it is really impossible to clarify the contributions of this paper step-by-step based on more closely-related existing literature. After all, after Section 2.5, there is effectively almost no citations. It is fine if the written content is really new and there is literally no related work, but I do not think so and this simply makes this paper hard to understand.  For example, I think this paper would be easier to read if the authors wrote this paper while referring to literature on POMDPs or Markov games, rather than formulating a completely new problem. Note that POMDP or Markov games are just examples. What I want to argue is that this paper will be much easier to understand if well-known notions that are closely related to this paper are properly referred after Section 2.5.\n- The authors seem to suppose that every reader fully understand Lang et al. (2024). I read through Lang et al. (2024) for reviewing this submitted paper, but I feel there are still leaps of logic in many parts of this paper. I would like the authors to properly introduce the related work as preliminary and write this paper in a more reader friendly manner.\n- My biggest concern regarding this paper is that it is unclear how to use the key idea or knowledge for scalable oversight. This may simply because I cannot fully understand this paper, but at least I think I can understand the main content of this paper. The current paper is too vague as a whole including theoretical analyses and examples, the reader cannot obtain useful knowledge from this paper. After all, while I don't think extensive experiments are necessary, I believe that conducting some experiments in a setting close to real-world applications (using small LLMs or API) would significantly enhance the reader's understanding of the true value of this paper.\n\n---\n- Lang, Leon, et al. \"When your AIs deceive you: Challenges of partial observability in reinforcement learning from human feedback.\" Advances in Neural Information Processing Systems 37 (2024): 93240-93299.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper address the problem so-called \"scalable oversight\" on how to supervise AI systems that is more intelligent than humans. This paper propose to model the human evaluator’s beliefs about how AI system behave, to better interpret the human’s feedback. The authors formalize human belief models and theoretically analyze their role in inferring human values, while discussing the inference ambiguity and its disappearing conditions. Finally, the authors propose using foundation models to construct covering belief models, providing a new potential approach to scalable oversight.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- First, I would like to ask the authors to write this paper in a more reader-friendly manner while referring to related studies. For example, I am wondering whether it is possible to frame this paper in the context of the game theory. On the other hand, if this paper is truly novel, it would be impossible to do so. For this point, I would like to refer to the comments by other reviewers or Action Editor. To be honest, I am not fully confident about the novelty of this paper.\n- Second, I recommend that the authors include some empirical proof-of-concept experiments to demonstrate the practical usefulness of their approach. Without it, I feel it is difficult for the readers to understand the true value of this paper. Given that this paper is highly theoretical, I don't think the authors need to conduct extensive experiments or benchmarking.", "broader_impact_concerns": "This paper is purely theoretical, so I do not see any ethical concern in this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)