Observation Interference in Partially Observable Assistance Games
Authors: Scott Emmons
Caspar Oesterheld
Vincent Conitzer
Stuart Russell
Venue: ICML 2025 poster (2025)
Keywords: assistance games
AI alignment
observation interference
partial observability
partially observable assistance games
Primary area: theory->game_theory
TL;DR: We study when an AI assistant would have an incentive to interfere with a human's observations.

Abstract: We study partially observable assistance games (POAGs), a model of the human-AI value alignment problem which allows the human and the AI assistant to have partial observations. Motivated by concerns of AI deception, we study a qualitatively new phenomenon made possible by partial observability: would an AI assistant ever have an incentive to interfere with the human's observations? First, we prove that sometimes an optimal assistant must take observation-interfering _actions_, even when the human is playing optimally, and even when there are otherwise-equivalent actions available that do not interfere with observations. Though this result seems to contradict the classic theorem from single-agent decision making that the value of perfect information is nonnegative, we resolve this seeming contradiction by developing a notion of interference defined on entire _policies_. This can be viewed as an extension of the classic result that the value of perfect information is nonnegative into the cooperative multiagent setting. Second, we prove that if the human is simply making decisions based on their immediate outcomes, the assistant might need to interfere with observations as a way to query the human's preferences. We show that this incentive for interference goes away if the human is playing optimally, or if we introduce a communication channel for the human to communicate their preferences to the assistant. Third, we show that if the human acts according to the Boltzmann model of irrationality, this can create an incentive for the assistant to interfere with observations. Finally, we use an experimental model to analyze tradeoffs faced by the AI assistant in practice when considering whether or not to take observation-interfering actions.

Observation Interference in Partially Observable Assistance Games
Scott Emmons" Caspar Oesterheld 2 Vincent Conitzer?
Stuart Russell
Abstract
1. Introduction
We study partially observable assistance games (POAGs), a model of the human-Al value align- ment problem which allows the human and the Al assistant to have partial observations. Motivated by concerns of AI deception, tively new phenomenon made possible by partial observability: would an Al assistant ever have an incentive to interfere with the human's obser- vations? First, we prove that sometimes an opti- mal assistant must take observation- interfering ac- tions, even when the human is playing optimally, and even when there are otherwise-cquivalen ac- tions available that do not interfere with observa- tions. Though this result seems to contradict the classic theorem from single-agent decision mak- ing that the value of information is nonnegative, we resolve this seeming contradiction by devel- oping a notion of interference defined on entire policies. This can be viewed as an extension of the classic result that the value of information is non- negative into the cooperative multiagent setting. Second, we prove that if the human is simply mak- ing decisions based on their immediate outcomes, the assistant might need to interfere with observa- tions as way to query the human's 's preferences. We show that this incentive for interference goes away if the human is playing optimally, or if we introduce a communication channel for the human to communicate their preferences to the assistant. Third, we show that if the human acts according to the Boltzmann model of irrationality, this can create an incentive for the assistant to interfere with observations. Finally, we use an experimen- tal model to analyze tradeoffs faced by the Al assistant in practice when considering whether not to take observation-interferin actions.
Assistance games provide a formalization of the human-Al value alignment problem (Shah et al.. 2020). They are based Hidden Goal MDPs (Fern al., 2014) and Cooperative Inverse Reinforcement Learning (CIRL) (Hadfield-Menell et al., 2016), an extension of Inverse Reinforcement Learn- ing (IRL) (Ng & Russell, 2000; Abbeel & Ng. 2004). In assistance games, a single human and a single Al assistant share the same reward function, but this reward function only known to the human; the assistant must learn it. In assistance games, desirable properties, such teaching by the human and learning by the assistant, emerge optimal solutions to the game (Shah et al., 2020). (This contrasts with prior work on algorithms where teaching is an explicit objective (Cakmak & Lopes, 2012; Goldman & Kearns, 1995; Balbach & Zeugmann, 2009).) For example, Wood- ward et al. (2020) find that deep neural networks solving an assistance game invent strategies that involve information sharing, information seeking, and question answering.
study qualita-
inneraatioonnnteeeerinng
Past analysis of assistance games was done assuming that the state of the world is fully observed by both the human and the assistant (Hadfield Menell et al., 2016; 2017). While Shah et al. 's (2020) definition of an assistance game allows for partial observability, they do not study its implications. In this work, we introduce the notion of a partially observ- able assistance game (POAG) to study the more general case faced in reality: when the world is only partially ob- servable. Partial observability raises new issues surrounding the communication of private information. A priori, we might hope that AI assistants never take any action that ob- structs information. Yet our analysis will show that even assistants which perfectly share our goals must sometimes obstruct information to communicate other, more important information.
This tension connects to broader work on AI deception, which recent research approaches from multiple angles. Park et al. (2024) provide philosophical definition and empirical survey of AI deception, while Ward et al. (2023) define deception in struetural causal games. Of particular relevance is work analyzing how reinforcement learning from human feedback (RLHF)-which can be seen as an algorithm for solving assistance games-can lead to decep- tion. Lang et al. (2024) prove that partial observability in
'quual contribution 'Center for Human-Compattbl AI, Univer- sity of California, Berkeley 'Foundations of Cooperative AI Lab, Carnegie Mellon University. Correspondence to: Scott Emmons <emmons @ berkeley.edu>>
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Observation Interference in Partially Observable Assistance Games
RLHF can create dual risks of deceptive inflating and over- justification. Complementing Lang al. (2024)'s theory, Wen et al. (2024) and Williams et al. (2024) provide ex- perimental evidence that optimizing for human feedback teaches language models to mislead humans. However, these works primarily focus on misaligned AI systems that deceive for their own goals. We study the subtle case where a perfectly aligned AI assistant might obstruct information for the human's 's benefit.
timestep games where players simultaneously act without observing the other players' actions. In our setting, the environment evolves over time, and the players can influ- ence each other's ss observations through their actions. Our results show that this influence on Dbservations-inctuding observation interference- a key feature that enables the communication of private information to achieve better out- comes.
In our setting, even if non-interference solution exists, it might require that the human send information to the assis- tant via an unnatural communication convention. We find that a second incentive for observation interference occurs if the human is instead just making decisions based on the immediate reward of those decisions. that case, the assis- tant's best response might require observation interference as a form of preference query (Section 5). We prove that this incentive for interference goes away if the human is playing optimally, or if we introduce communication channel for the human to communicate her preferences to the assistant.
Concretely, we seek to understand whether observation in- terference emerges as optimal behavior in an AI assistant that shares the human's 's goals. We take a game-theoretic approach, studying qualitative properties of optimal policy pairs and best responses in POAGs. To start, we define an observation interfering action as one which provides the human with subset of the information available with otherwise-equivalle
action. We then analyze if the Al as- sistant ever takes observation interfering actions in optimal policy pairs or best responses.
Our analysis reveals three distinct incentives for an Al assis- tant to take observation interfering actions. First, when the assistant has private information, it might need to interfere with observations to communicate its private information to the human (Section 4.2). This can happen even when the human is playing optimally, and even when there are otherwise-equivalent actions available that do not interfere with observations. This result presents a puzzle, as it seems to contradict the classic theorem from single-agent decision making that the value of information (sometimes also called the value of perfect information) is nonnegative (e.g., Koller & Friedman, 2009, Sect. 23.7; Russell & Norvig, 2010, Sect. 16.6.3). To resolve this seeming contradiction, we develop notion of interference defined on entire policies rather than individual actions. While optimal solutions (i.e., human-Al policy pairs) might involve the Al assistant taking individual actions which would on their own be observation interfer- ence, we prove that there is always an optimal solution with no observation interference when we consider the Al assis- tant's overall policy. This can be viewed as an extension of the classic result that the value of (perfect) information is nonnegative into the cooperative multiagent setting.
When the human is making irrational decisions, it creates a third incentive for the assistant to interfere with observations. For example, we show that if a Boltzmann-rational decision maker has a higher error rate when presented with complete information, the assistant might suppress information to give the human an easier decision (Section 6).
Finally, in Section 7, we use an experimental model to inves- tigate tradeoff's the assistant faces when deciding whether or not to interfere with observations. In line with our theory, we find that observation interference allows the Al assis- tant to communicate private information, but it comes at the cost of destroying useful information. Measuring this tradeoff, we find that having more private information leads to a stronger incentive to interfere with observations.
2. Preliminaries/ / Setup
2.1. Partially Observable Assistance Games
We study partially observable assistance games (Shah et al., 2020):
Definition 2.1. A partially observable assistance game (POAG) M is a two-player DecPOMDP with human or principal, H. and an Al assistant, A. The game is described by a tuple, M = (S, (AH AAA),T( ), (O, R(.,'ri-)), (1H (2A,0((, (...)).....).............)))).)) P, ). with the following definitions: S. a set of world states: S; AH, a set of actions for H: o E AH, AA, a set of actions for A: a E AAA T(- .,.."), conditional distribution on the next world state, given previous state and action for both players: T(s' | s, s, )aA); θ a set of possible static reward parameter values, only observed by H: 0 E θ R(-.....)), a parameterized reward function that maps world states, joint actions, and reward parameters to
This result connects to a broader literature on the value of in- formation in multiagent settings. In games with competing interests, it is well-known that introducing common knowl- edge can lead to worse outcomes for all players (Kamien et al., 1990). Using a set-theoretic framework, Bassan et al. (2003) establish a class of general-sum games where addi- tional information Pareto-improves all of the Nash equilibria. Their class of games includes common- -ayooff games. Using a probabilistic framework, Lehrer al. (2010) extend this analysis to alternative solution concepts. Notably, Bassan et al. (2003) and Lehrer al. (2010) consider only single-
2

---

Observation Interference in Partially Observable Assistance Games
real numbers: R: S x AH x AA @ - RR; RR QH, set of observations for H: OH E H ΩA, set of observations for A: oA E ESAA O(..- ...,)), conditional distribution on the observations, given the next world state and action of
H knows A A's policy along with the POAG specification M, then H can form calibrated beliefs about the world state. For any timestep and state St, H can form P(st of the probability of St given H' s''s observation history oi:-
Ht
both players: ( _AA s',aH, ,,H Po(ss)) ). a distribu- tion over the initial state, represented as tuples: Po(sp;0) 0); and 7, a discount factor: 7 E [0.1].
In an iterated setting where A updates its policy between iterations, H can form beliefs if H additionally knows the policy update rule.
We denote H's and A's marginal observation distributions as (oH| s'.a a aA) = =Lon on O(oH AA | s', ss aA and O (oA s' H. aA = oH O(oH, .A s', ss aa We consider H policies nh which, at timestep i, take as input the full history of H's observations and actions h, E (S2H x y and map to distribution over ac- tions AAH. A's policy x² : (SAA x A + AA is analogous. We call H a best response to T when H maximizes expected discounted reward given TA, i.e.. T E arg max;" E,H, x [E== 7 R(s,, af a. [0]] where the expectation is taken over trajectories induced by the policies (xr x A and initial distribution Pa- The best response for A is defined analogously. A pol- icy pair (TH A is optimal if it maximizes the ex- pected discounted reward in the POAG: (T A) =
Proposition 2.3. Suppose A is updating its policy each iteration of the game. Knowledge of the game dynamics, of A 'ss initial policy, and of A's ''s update rule is sufficient for H to form calibrated beliefs about A A's future policy and of the world state.
Remark 2.4. Propositions 2.2 and 2.3 hold even if A is interfering with observations (Definition 3.2).
Remark 2.5. Proposition 2.2 and Proposition 2.3 continue to hold if only knows a prior over A 's policy. H can form a posterior using Bayes' rule; the posterior is calibrated if the prior is calibrated.
When H knows A as policy, the preceding results show that H can form calibrated beliefs about the world, even when A is interfering with observations. Observation interference in- creases H'S 's uncertainty, but it doesn't It break the calibration of H's beliefs. Because H can still form calibrated beliefs in this setting, our work uses the concept of "interference" rather than the concept of "deception."
arg max EAH 2 af. a [0]].
Note that optimal policy pairs are in particular Nash equi- libria for the shared reward function R. Computationally, POAGs are equivalent to 2-player decentralized partially ob- servable Markov decision processes (DecPOMDPs). Thus, finding optimal policy pairs for POAGs is NEXP-hard in general (Bernstein et al., 2002) (cf. Reif, 1984). A POAG may have multiple distinct optimal policy pairs, as there may be different ways for H and A to communicate or resolve coordination problems.
3. Defining Observation Interference
Observation Interference First, we define what interfer- ence means. Intuitively, interference is taking action so that the human receives a less informative signal about the state. In particular, the human receives, in some sense, a subset of the information. We formalize this by saying one signal is less informative than another about the state if (without knowing the state) we could generate signal from the other (cf. Blackwell et al., 1951; Blackwell, 1953; de Oliveira, 2018).
While the examples in this paper are simple, POAGs-an thus all our positive -inuts- the broad generality of DecPOMDPs. POAGs can model games where H acts first, where A acts first, or where H and A act simultancously. POAGs allow both H and A to observe private information at multiple times, as well as take actions that influence both the state of the world and each other's observations.
Definition 3.1. Let (P(- ss)sss and (P(. s)ses be families of probubility distributions over Ω. We say that P is at most as informative as P if there exists a stochastic function F: Ω ~4 Ω (mapping observations to random variables over observations) s.t. for all states s we have F(X)= ~ P(. if P(. 8). We say that P is (strictly) more informative than P if P is at least as informative as P but not vice versa.
2.2. Beliefs and Calibration of Beliefs
We are motivated to study observation interference because of its potential impact on H's belief about the state of the world. If A interferes with observations, could this cause H to have false beliefs?
To address this question, we apply known techniques to es- tablish what information H needs to form calibrated beliefs in a POAG. (See Appendix A for proofs.) The simplest case of H knowing A'S 's policy is when A is playing a fixed policy:
Why do we include the condition "for all states 8 in Defini- tion 3.1? Intuitively, we want it always to be possible to use the stochastic function F to reconstruct the less informative signal from the more informative signal. Since our setting is partially observable, the "for all states s" condition allows a
Proposition 2.2. Suppose A is playing a fixed policy. If
3

---

Observation Interference in Partially Observable Assistance Games
player do this reconstruction in any scenario, even if their observations don't enable them to infer the state.
1mm=m')O(oH, o 8 , K,,H,aA,,, aAA) For everything else, the messages are simply ignored.
With this definition in hand, we define an observation- interfering action as one that results in the human's 's ob- servation being less informative about the state than the observation distribution resulting from another assistant ac- tion. We additionally require that this other action has the same effects on the state and immediate reward. After all, it is clear that sometimes A has trade off providing infor- mation to H with optimizing its effect on the environment.
Plausible Human Policies We may have various expec- tations on how H will play in a POAG. Especially if there are multiple optimal policy pairs, we may expect some of these policy pairs to be more plausible because they require simpler behavior of the human (ef. Hu et al., 2020; Treutlein et al., 2021). Both of the conditions below are based on the idea that A and H are unlikely to use consequential actions in the world to communicate with each other.
Definition 3.2. Let M be any POAG. We say that âA is observation-interferigg if there exists some other action a s.t. â^ and aA have the same effect on state tran- sitions and immediate rewards, but for all a we have that (OH(. | aH, 8, (,,,, )ses is more informative than (OH(- | ,, ))s,s
Our first condition intends to express a form of naivete on H's part in how she interprets her observations. Roughly, the condition says that H takes her observations at face value, i.e., as if they were not interfered with. She does not try to interpret them
form of communication by A. For instance, if H reads thermometer as saying that temperature is 37 degrees, she chooses under the assumption that the temperature is indeed 37 degrees, rather than, say, interpreting 37 as a message sent by A.
To discuss policies that play observation-interferin actions, we use the following definition:
Definition 3.3. We say that a policy A interferes with observations at the action level (or equivalently, takes observation-interfeent
Definition 3.6. We say that a human policy H. observes naively if n H is a best response to some V that does not interfere with observations at the action level.
actions) in a POAG M if there is any history h E (ΩA x A where x A- (- h) assigns positive probability to an observation- inttefrinnnnteeeeriinn action.
The second property is that when the human knows that her action has no effect on the state, then she chooses among actions that maximize immediate reward. To state this formally, we first define the following. We say that in h actions don't affect state transitions, if for all s s.t. we have P(s hi ,T) > 0 for some TA, have that for all a the transition probability P(s' | a ,at)) is constant over a We say that π" myopically maxi- mizes reward in h if there is some distribution a e A ((AA) s.t. n (. in arg max, EaA A P [R(s, aH aa 0)). (Intuitively, a is H's belief about what action A is go- ing to take.)
Lack of Private Information To understand the condi- tions under which interference occurs, it is useful to consider POAGs where one of the players has no private information.
Definition 3.4. For a POAG M, we say A has no private information if there exists a function I determining A's observations from H's observations. For all state-action tuples (s', a H A and observation pairs Ho A²) E supp(O(s, . s',, ",aH, (cH,a))),, )) then f must have f(o = AA.
IH) randomizes only over actions
Communication To further understand the motivations behind interference, we will also consider POAGs in which the players are able to directly communicate. Thus, for any given POAG, the following defines variant of that POAG in which the players have an additional channel for communication. We will always assume that the channel has enough bandwidth for the sender to share all private information, i.e., that there is an injection from the sender's observation space into the message space.
We say that a human policy π acts naively if whenever H faces a choice that doesn't It affect state transi- tions (but potentially affects A 's observation). H plays an action that myopically maximizes reward.
Definition 3.7.
Importantly, if H acts naively, she is unwilling to play a suboptimal action to communicate information to A.
Definition 3.5. Let M be a POAG. Define MA-H H and M H as a variants of M with unbounded communication channels. We define H-A below;
4. Communicating Private Information is an Incentive for Observation Interference
MA-H and MH++ are analogous.
To construct M AH+A, let M be some set of possible mes- sages/signals s.t. there is an injection S2A + -M. Then, con- struct a new human action space AH =AH x -M and new assistant observation space Ω = Ω x M. The new ob- servation kernel has ò (o,, (oA m²) 8', (at m)) a)
4.1. Revealing Errors can Emerge as an Optimal POAG Solution
We can model RLHF within the POAG framework as fol- lows: A's goal in RLHF is to satisfy H's preferences. In a POAG, this corresponds to the shared reward function

---

Observation Interference in Partially Observable Assistance Games
which has a parameterization 0 that only H knows. In RLHE, A rolls out trajectories, and H picks which trajectory is pre- ferred. A POAG can model this by letting H observe pairs of trajectories explored by A but only giving H a binary action (to choose which trajectory H prefers). In RLHF, A's final policy maximizes an estimate of R. based on a dataset of H's preference comparisons (Lang et al., 2024, Proposition 4.1). In the POAG framework, A can compute this policy based on A' 's observations of H's binary actions.
own private information to H. Consider the following ex- ample.
Example 4.3. H has typed apt 1ist -a cuda to the list of cuda versions available to be installed. Out of 10 total versions, only a (non-empty) subsel are available. And of these available versions, only a subset are compatible with the other environment software.
First, A takes an action. For each of the 10 total cuda versions, A can choose to not suppress from the list of available packages. This gives A 2n total actions, where I action is non con-observation interference (suppressing nothing), and the remaining 210 1 actions interfere with observations.
Past work has shown how RLHF can cause misleading (Wen et al., 2024) and deceptive (Williams et al., 2024: Lang et al., 2024) behaviors. Specifically, Lang et al. (2024, Example B. show that in order to get better human feedback, RLHF can have an incentive to hide error messages.
Second, H takes an action. H has 10 possible actions which try to install the corresponding version of cuda if it appears in the version list. If an available cuda version that is compatible with the other environment software is installed, it yields +1 reward. Otherwise, it yields O reward.
In contrast RLHF, we show with the following example that revealing error messages can emerge in POAG solu- tions.
Example 4.1. First, A is executing on a remote machine where logging has been disabled by default. A takes one of two actions: (1) Attempt install cuda. The installation succeeds with 50% probability. An empty observation is produced (since logging is disabled). (2) Re-enable logging and attempl to install cuda. The installation succeeds with 50% probability. An observation is produced containing a success or failure message.
Suppose A sees which versions are compatible with the other software in the environment, but H doesn't. 't. Then A optimal policy is to suppress the versions of cuda that are incompatible.
Our high-level takeaway from this example is that in some POAGs, all optimal policy pairs require A to take Observationniiterferii actions. Importantly, in the optimal policy pair for the above example, H observes naively. In particular, the above doesn' 't require H and A to have some communication protocol and for H to interpret her observa- tions as encoding A belief'. H can act as if no interference is happening. We thus summarize the high- takeaways in the following result, with details in Appendix B.3.
Then, H takes one of two actions: (1) Run an experiment. If cuda is installed successfully, this yields +1 reward. Other- wise, it yields- -2 reward. (2) Don't 't run an experiment. This always yields 0 reward.
In the optimal policy pair, A reenables logging; this reveals errors to H!
Proposition 4.4. There exists POAG M where all optimal policy pairs (x HH) have that n interferes with obser- vations at the action level and that h observes and acts naively.
In fact, if A has no private information, then it never needs to take iiservation-interfriiii actions for an optimal solution!
Theorem 4.2. Let M be any POAG. Let A have private information. Then there is an optimal policy pair (n n A for M in which ' does not interfere with observations at the action level (and H observes naively).
Intuitively, in Example 4.3, A interferes in order to convey information to H. A knows H's optimal choice, but cannot tell her. So, A needs to interfere in way that leads H to the optimal choice.
4.2. Communicating Private Information is an Incentive for Observation Interference at the Action Level
The need for A to take observationninterfering actions to communicate to H disappears if A has other means of communication. For instance, if in Example 4.3, A could simply tell H what to do, then A wouldn't 't need interfere. To formalize this intuition, we now prove that if A can communicate with H. then there is always an optimal policy pair that does not require interference.
One might hope that A would never take observation- interfering actions. After all, classic theory tells us that when H is in single-agent setting, the value of (perfect) information nonnegative (e.g., Koller & Friedman, 2009, Sect. 23.7; Russell & Norvig, 2010, Sect. 16.6.3): more informative observations never lead to worse solutions. But as it turns out, when H and A interact, there are some POAGs in which all optimal policy pairs require A to take observation incervationniteeferiing actions. The main reason for A to take inservationinnteferrin actions is to communicate its
Theorem 4.5. Let M be any POAG, and provide A with an unbounded communication channel to H. forming A Then there is an optimal policy pair (TH a for M MA-H HH where A does not interfere with observations at the action
5

---

Observation Interference in Partially Observable Assistance Games
level and H observes naively.
suppressing versions "tampering at the action level."
The key difference with Definition 4.6 is that H knows A 's policy. Suppose that A's policy π is to suppress exactly the versions of cuda that are incompatible with the other software in the environment. Because H knows that A suppressed the incompatible cuda versions, seeing the fil- tered list tells H which versions of cuda are compatible! Although suppressing versions is strictly less informative under Definition 3.2 (when H doesn't 't know A's 'ss policy), suppressing versions provides H with new information un- der Definition 4.6 (when H knows A's policy). Accordingly, T A is interfering with observations at the action level but not at the policy level.
One could argue that in practice, an unrestricted commu- nication channel between A and H could usually be made available. However, Theorem 4.5 ignores various real-world obstacles. For one, it considers communication that incurs no cost, but in reality, communication costs H time and effort. Second, the optimal policy pair requires A to send information in a way that H can reliably understand and act upon. We expect that in practice, A and H sometimes can- not understand each other. Therefore, despite Theorem 4.5, we think observation interference is of broad practical rele- vance, even where A can, e.g., send text messages to H.
4.3. Optimal Policy Pairs Never Require Observation Interference at the Policy Level
In our examples, we will mostly consider actions that in some sense act directly on H's observations. Yet Defini- tion 4.6 also considers the informational effects of physical actions. For example, if A (visibly) tries to open a door that A knows to be locked, then this reveals to H that the door is locked. Consequently, not trying to open the door (when A knows it to be locked) is an instance of observation interference in the sense of Definition 4.6. While having the same (null) effect on the state of the world, trying to open the door provides H with more information.
In Definition 3.2, we first define observation interference as a feature of actions. We then say in Definition 3.3 that a policy interferes with observations at the action level if and only if it ever takes an abservation-interferiii action.
Because the definition is ultimately about actions, it doesn't 't consider how π might choose to take observation- interfering actions in way that depends on A's observa- tions. To account for A dependence on its observation, we define an alternative notion of what it means for a policy to interfere with observations.
As in Example 4.3, cases which appear to destroy informa- tion when viewed at the action level may actually provide new information when viewed at the policy level. In fact, the following theorem shows that it's never strictly necessary to interfere with observations at the policy level.
Let Pot be the distribution over human observations at time t. Further, let LT ^T) be the set of possible states at time t.
Definition 4.6. Let M be a POAG. We say that A's pol- icy a interferes with observations at the policy level if there exists some other partial policy of for time step t s.t. at and ot have the same effect on state transi- tions and immediate rewards, but for all H we have that Port (- | HH st+1; #0zt "=)))==) S s6.) is less in- oes
Theorem 4.7. Let M be any POAG. Then there exists an op- timal policy pair ( H ) M s.l. A does not interfere with observations at the policy level.
This contrasts with Proposition 4.4: whereas it is sometimes necessary to interfere with observations at the action level, it is never necessary at the policy level.
formative than the corresponding distribution if we replace To:t with (# (# 7
The main idea behind this proof is similar to the proof of Theorem 4.2 (given in Appendix B.3). That is, if we start with an optimal policy in which A buseevation-inteeffees, then we can replace A A's policy with the corresponding more informative policy and update H's policy to imitate the garbling. The proof of Theorem 4.2 considers the set of actions, which is finite. The main extra difficulty in proving Theorem 4.7 is that we must deal with spaces of policies, which may be infinitely large. Thus, if we replace a policy with more informative one, there might be new policy which is even more informative, and so on forever.
Compared our previous action-level notion of observation interference (Definition 3.2), this new policy-level notion (Definition 4.6) differs in how it treats H's inference process. Whereas the action-level notion models inference about isolated observations, the policy-level notion allows H to make inferences in the context of A's 's overall strategy.
We now revisit Example 4.3. For observation tampering under our earlier Definition 3.2, H simply knows that A has taken the action to suppress some versions of cuda; H does not know anything about A 's policy. For all H knows, A 's policy could be to randomly suppress cuda versions or to always suppress the same cuda version. Thus, suppressing any version is strictly less informative for H than the list of all available versions. This is why Definition 3.2 calls
Note that there are many possible ways to extend or refine Definitions 3.2 and 4.6 in ways that preserve our key results. We choose Definitions 3.2 and 4.6 in part for their simplicity; for more discussion of this point, see Appendix G.
6

---

Observation Interference in Partially Observable Assistance Games
5. Querying H's Preferences is an Incentive for Observation Interference
interfere. In particular, even if A does not interfere and the two available nodes are CPU-optimized, H may sim- ply communicate her CPU-versus-GPU preference anyway! That is, when facing a choice between CPU-optimized node and 2, she may choose, say, 1 if she favors GPU-optimized nodes and if she favors CPU-optimized nodes. However, this type of human strategy seems implausible, as it would require H and A have settled on some communication strategy that overrides H's immediate preferences about the machines that H can in fact choose between.
We now study second reason A can have for interfering with observations. We have already shown (Theorems 4.2, 4.5 and 4.7) that even H has private information and no communication channel, there's ' always an optimal policy pair in which A does not interfere, as long as A doesn't have private information. So. if H plays best response to A 's policy, then A can choose non-interference policy without loss of utility. However, if H does not play best response A, then reasons for interference emerge that are more subtle than those in the A + H case.
The key point of Example 5.1 is thalewhile there is some optimal policy pair without observation interference- there is no plausible optimal policy pair that avoids observation interference. More specifically, we use the notion of acting naively (Definition 3.7) express this notion of plausibility and rule out the above policy. We thus obtain the following proposition (with proof in Appendix C): in some POAGs, if we want to play an optimal policy pair and we want H to be able to act naively, then A has to interfere with observations.
Intuitively, A might need to interfere with observations to elicit H + A communication. Suppose A needs some information from H. but H is acting naively (see Defini- tion 3.7) in way that does not reveal her private infor- mation. By changing H's observation, A make H's naive response communicate useful information to A. The following example illustrates this phenomenon.
Proposition 5.2. There is a POAG M with the following properties. For every optimal policy pair (T ^^), at least one the following holds: (i) π is not acting naively, or (ii) π interferes with observations at both the action and policy levels. Additionally, there exists an optimal policy pair (T ; where nH acts naively and T interferes with observations at both the action and policy levels.
Example 5.1. H would like to schedule a j.bob on a clus- ter. She can choose between two nodes. By default, she receives a signal from the environment about the two nodes' specifications. Each node may be either GPU-optimized or CPU-optimized. Also, the CPUs may be either AMD or Intel.
H has a strong preference between GPU-optimized and CPU-oplimized nodes. She has a weak preference between AMD and Intel. These preferences are unknown to A.
These properties continue to hold if we require that in M, A has no private information or can arbitrarily send messages to H (i.e., there is POAG M s.t. M MA-H..
A can interfere with H's observation about the available nodes. In particular, A can make it so that choice between two CPU-optimized nodes appears choice between GPU-optimized and CPU-optimized node. A observes H''s 's choice. Later, A is charged with scheduling a job H and has to choose between a CPU- and a GPU-optimized node on H's behalf.
Intuitively, the problem in the above example is that the human has private information that she needs to communi- cate with her choices. (Because her choices yield different immediate rewards, naive choices fail to communicate.) As before, the need for interference or non-naive choice disappears if the human has no private information pro- vide. The following shows that the need for interference / non-naived also disappears if H can communicate with A. To also rule out the need to interfere with observations for A + H communication (discussed in Section 4.2) we assume communication channels in both direction.
If chooses naively upon seeing only CPU-optimized nodes (simply choosing her favorite), then A best response in- terferes with observations at both the action and policy levels. Interfering with observations allows A to learn H's preference about GPU- vs CPU-optimized nodes.
Theorem 5.3. Let M be POAG. There exists an optimal
policy pair (x. x for HesA where I is naive and assumes honesty while A does not interfere at either the action or policy levels.
In Example 5.1, one might ask why A can't 't just ask H each time A makes a decision. Simply asking H's preference is reasonable when has only one decision to make. However, we are motivated by cases where A has many decisions to make, and asking H's preferences each time would be cumbersome.
6. Human Irrationality is an Incentive for Observation Interference
At first sight, Example 5. may appear to be a counterex- ample to Theorem 4.2, which states that a non-interfering optimal policy pairs exists. However, note that Example 5.1 actually does have optimal policy pairs in which A doesn't
Finally, we consider third reason for observation interfer- ence: human irrationality or bounded rationality. Roughly, reducing the amount of information supplied to the human may simplify the human's 's decision problem and thus im-

---

Observation Interference in Partially Observable Assistance Games
prove her decision making. Importantly, this motivation for observation interference may exist even if neither H nor A has any private information.
7. Experiments
Motivated by the theory in Section 4 and Section 6, we develop model game and run experiments to analyze:
As our model of human decision making, we adopt Boltz- mann rationality (Luce, 1959; McFadden, 1973), which has recently been used in (C)IRL (Laidlaw & Dragan, 2021; Ramachandran & Amir, 2007; Ziebart et al., 2008).
1. How does the amount of H's irrationality affect A 's incentive to take observation-interfrrig actions?
Definition 6.1. Let M be a POAG. Let π A be A policy in M. We say that H's policy n. is a Boltzmann-ration response to π if there exists some 3 > O s.I. for every human observation history ht that arises with positive prob- ability in M under (T T we have that n. (a h) x
2. How does the amount of A's 's private information affect A 's incentive to take observationninterfering actions?
7.1. Experiment Details
exp AIE 1" R(S A, A π TT
We study a game where selecting the best action requires combining private observations known only to and pri- vate observations known only to A. The game presents A with a tradeoff: A can interfere with observations to com- municate information that only A observes, but interfering also destroys information that only H observes.
turns out that even if the Boltzmann-rational human has calibrated beliefs, A's 's optimal policy sometimes interferes with observations, even if neither A nor H has private in- formation. Intuitively, providing more information may sometimes result in less clear-cut decisions, i.e., decision situations with a smaller difference between the correct and incorrect option. To illustrate this phenomenon, consider the following example.
Concretely, the game has d products. Each product i has two attributes, H, and R,, drawn i.i.d. from Uni 1). Each product's utility the sum of its attributes, U, = H, + RG. The game consists of two moves. First, A sees R for 1,.... k where k is the number of A's private observations. A chooses set of products to interfere with. For the products A interfered with, H sees H, = -oo; for the remaining products, H sees H, = H4, Second, H chooses a product 0; Both H and A receive a common payoff of the chosen product's utility, U,,
Example 6.2. H is ranning a terminal command and is unsure whether to run the command with flag I or flag 2. With equal probability, either flag 1 or flag 2 is better, and how good the flags are differs by either a little or lot. The worse flaseflag always yields a utility of 0, while the better flag either yields a utility of I or a utility 7. Thus, H is uniformly at random in one of four states. A has two actions: man and t tidr. The man page is a long document that tells the human exactly what the values of the flags are (i.e., the exact state: which flag is better and whether its utility is I or 7). The tidr page is short summary that tells the human which flag is better, but not by how much (i.e., ruling out half the states, leaving half remaining). Thus, the expected utiitty of the better flag is (and of the worse flag is 0).
We assume the human's product selection policy is Boltz- mann rational over their observed values H,:
Definition 7.1. H's Boltzmann selection policy chooses products by Boltzmann distribution over H4, the observed product values: x <a.)) exp(3H.). The parameter 3 controls H's rationality.
Intuitively, both the t tidr and man pages allow the human to choose optimally, but the man page is more complicated and therefore more likely to be misinterpreted. Choosing specific utilities, the effect of interference under Boltzmann rationality is as follows. If A interferes (i.c., provides the eldr page), then H always chooses between a utility of 4 and 0. If A does not interfere, then half the time, H chooses between utilities and 0, and half the time H chooses between utilities 7 and O. It turns out that for achieves higher utility in expectation under the condition where A interferes. Building on this idea, we can prove the following (with details in Appendix D).
We consider A policies that always interfere with ic obser- vations for some fixed k. Call these policies k-interference. We study the optimal such policies, characterized by the following result:
Proposition 7.2. Consider policies that always inter- fere with observations for some fixed k. Among the interference policies for given k. A ''s best response H's straightforward product selection policy is as follows. A interferes with the k smallest RG values where R, = R, if A observes R2 and R; = 0.5 otherwise.
1, H
Proposition 6.3. For every 3 > 0, 3 a POAG in which nei- ther H nor A has private information s.t. all 3-Boltzmann- rational/optimal policy pairs (х x²) have Λ interfere with observations at both the action and policy levels.
We consider a game with d = 5 products. We vary H's number of interferences & E {0, 2, 43. We run a Monte Carlo simulation with 30,000 trials to calculate the expected payoff in each setting.
8

---

Observation Interference in Partially Observable Assistance Games
Number of Interferences
1.30
www.
Expected Reward
1.25
Expected Reward
1.20
1.2
1.15
Number of Private Observations
1.10
1.1
1.05
1.00
1.0
10
10
10 10a
Rationality Beta
Number of Interferences
(a) A's Number of Private Observations =
(b) H's sss Rationality Coefficient 3 = 0c
Figure I: Incentives to interfere with observations in the product selection game. (Left) When H is highly irrational, it's best for A to interfere, effectively making the choice for H. As H becomes more rational, there an increasing cost to interference, and there's tradeoff: A should interfere to communicate some information, but not destroy too much information by excessive interference. (Right) In line with Theorem 4.2, A has no incentive interfere when A has no private observations. With more private observations, A has more incentive to interfere.
7.2. Varying H's Rationality
of observation interference k change with the number of private observations available to A.
How does H's rationality impact A's incentive for obser- vation interference? We fix A to have 2 private observa- tions. We do a logarithmic sweep over H's rationality coeffi- cient 3 E {0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30, 1003. Figure la shows how the expected reward changes w.r.t. ,3.
Figure lb shows how the expected reward changes depend- ing on k; the number of interferences. When A has no pri- vate observations, then reward decreases for each increased number of interferences. However, the number of A private observations increases, the relative ordering of the observation interference policies changes; with more pri- vate observations, A has an incentive to interfere with more observations. This confirms our hypothesis based on The- orem 4.2. Nevertheless, there is a limit to A observation interference incentive. Because interfering with observa- tions destroys H's information, A must be careful not to interfere too much
When H is highly irrational at 3 0.01, A should interfere with as many sensors as possible, effectively choosing H's action. When H is acting little better than randomly, it's best for A to choose H's action, even when A has less informa- tion than H. For larger values of B3, a tradeoff emerges. As A has two private observations, there is an increasing bene- fit to interfere to communicate information H. However, as H can now make use of their own private observations, A must be careful not destroy too much of H's private information by excessive interference.
8. Conclusion
Limitations and Future Work Optimal policy pairs sometimes require H and A to have a shared communi- cation protocol (e.g., Example 5.1). would be interesting to study additional solution concepts, such as correlated equilibria and communication equilibria, to handle this sort of communication (Forges, 1986). While we consider only a single human and single assistant, it would also be inter- esting to study scenarios with multiple humans and multiple assistants. As we focus on the Boltzmann rationality model of human decision making (Definition 6.1), future work could consider other human models and empirical valida- tion with human subjects. Lastly, while run experiments in one model of POAG, it would be interesting to see if and how our experimental trends generalize to other POAGs, including POAGs where must query H's preferences.
7.3. Varying A's 's Private Information
How does the amount of private information available to A influence A's 's incentive for observation interference? In Theorem 4.2, we showed conditions under which private observations for A are a necessary condition for observa- tion interference to occur. Now, we analyze the degree to which private observations incentivize observation interfer- ence. Based on Theorem 4.2, we hypothesize that there are cireumstances where more private information leads to more observation interference.
We vary R's number of private observations in {0. 1, 2, 3,555). 5). We consider A's k-interference policies and analyze how the relative performance of different levels

---

Observation Interference in Partially Observable Assistance Games
Impact Statement
tions is easy. We believe our characterizations remain useful even in complex environments (where we can't expect opti- mal policies), although we can" ' make as definitive claims as we can about optimal policies.
Al assistants are being developed and deployed in settings where humans can only partially observe what's happen- ing. For example, Al assistants including ChatGPT, Claude, and Gemini can search the web while only returning sum- maries to users (OpenAI, 2024; Anthropic, 2025; Google, 2024). Moreover, the sorts of AI models powering these assistants are processing increasingly long inputs. Whereas the original ChatGPT model could only process 4096 in- put tokens, today's ' Gemini 1.5 Pro can process 2,000,000 input tokens-whicht is roughly 100,000 lines of code, or 16 novels of average length in English (Google, 2025). In the future, we anticipate that AI assistants will be deployed at increasing scale, independently taking more actions on behalf of users and processing increasingly long context lengths. We thus expect that over time, humans will have less and less ability to directly observe everything that's happening.
We have discussed allowing communication between H and A. A complexity-therre H and A share all private information, the game effectively turns from DecPOMDP into a POMDP. Solving POMDPs is PSPACE-complete and thus likely easier than solving DecPOMDPs.
argument favors this solution:
Author Contributions
The project was conceived by S.E., who developed the ini- tial theorems and examples. C.O. led the theoretical de- velopment, proving the main results and formalizing the examples. S.E. conducted the experimental analysis. The manuscript was written collaboratively, with S.E. leading the introduction, experimental, and conclusion sections, while the theoretical sections were co- co-written by S.E. and C.O. V.C. and S.R. advised throughout the project.
Even when the AI assistant and the human have perfect value alignment, we show how observation interference can emerge from several distinct incentives. As we focus on optimal assistants-analyyiinn optimal policy pairs and best responses-all all of the incentives for observation interference that we consider are done for the human's benefit. This creates a nuanced picture, suggesting that not all observa- tion interference is inherently bad. As Al assistants might exhibit observation interference for mix of good and bad reasons, it would be interesting for future work to explore how to handle this nuanced situation. For example, future Al systems could be designed with transparency about when interference occurs and user controls to override interference when desired.
A Acknowledgments
The authors thank Mark Bedaywi, Emery Cooper, Anea Dragan, Andrew Garber, Linus Luu, and Rohan Subramani for helpful discussions. C.O.'s work is supported by an FLI AI Existential Risk Fellowship. V.C. acknowledges finan- cial support from the Cooperative Al Foundation, Polaris Ventures (formerly the Center for Emerging Risk Research), and Jaan Tallinn's dovor-ddvised fund at Founders Pledge. S.E. and S.R. are grateful for Open Philanthropy's gift to the Center for Human Compan-Co AL.
With this theory, our goal is to understand the causes of observation interference and help disentangle them in prac- tice. We intend for our work to help AI developers build assistants that their users can trust. Our work is primarily theoretical, and we are not aware of any ways it could be used to cause harm.
References
Abbeel, P. and Ng, A. Apprenticeship learning via inverse reinforcement learning. In ICML, 2004.
Anthropic. Claude can now search the web, 2025. URL
http://wwwwattrroc..ooo com/news/ web search.
Computational Complexity Given that finding optimal policies in POAGs is NEXP-hard hard, how might our results ap- ply in a given environment? Most of our paper is descriptive, characterizing when observation tampering could happen. Complexity considerations could affect these results in ei- ther direction. It's easy to construct environments where finding good inpervation-inteferinn policies is computa- tionally intractable but constructing good non-interfering policies is easy; and vice versa. In practice, complexities of the environment can be orthogonal to incentives to interfere. For instance, a real-world version of the CUDA example is complex (A assesses complicated software compatibility issues), but the decision whether to interfere with observa-
Ash, R. Information Theory, volume 19 of Interscience Tracts in Pure and Applied Mathematics. John Wiley & Sons, 1965.
Balbach, F. and Zeugmann, Recent developments in algorithmic teaching. In Language and Automata Theory and Applications. Springer, 2009.
Bassan, B., Gossner, O., Scarsini, M., and Zamir. S. Positive value of information in games. International Journal of Game Theory, 17713117331, 2003. ISSN 1432- 1432-1270. doi: 10.11077/000880030114
10

---

Observation Interference in Partially Observable Assistance Games
Bernstein, D. S., Givan, R., Immerman, N., and Zilber- stein, S. The complexity of decentralized control of Markov decision processes. Mathematics of Operations Research, 27(4):819- 2002. doi: 10.1287/moor.2.. 4.819.297. URL https: /dl. acm...c.mcom htt.m./wrg/oo////o////oo/o 1287 /moor.ooo.... .2w.x.7..2. .4. 919.297.
26th International Joint Conference on Artificial Intelli- gence, IJCAI pp. 220-227. AAAI Press, 2017. ISBN 9780999241103.
Harris, C. R., Millman, K. van der Walt, S. Gommers, R., Virtanen, P., Cournaeeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M. H., Brett, M., Haldane, A., del Río, J. F., Wiebe, M., Peterson, P.,
Blackwell, D. Equivalent comparisons of experiments. The annals of mathematical statistics, pp. 265 265-272, 1953.
P., Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H., Gohlke, C., and Oliphant, E. Array programming with NumPy. Nature, 585(7825) September 2020. doi: 10. 22266441566-00022499-2 2649-2. URL https:///oi..om 1 10 1001008///0 4 00111166600024--22
Blackwell, D. et al. Comparison of experiments. In Proceed- ings of the second Berkeley symposium on mathematical statistics and probability, volume 2, pp. 93-102, 1951.
Cakmak, M. and Lopes, M. Algorithmic and human teach- ing of sequential decision tasks. In AAAI, 2012.
Hu, H., Lerer, A., Peysakhovich, A., and Foerster, J. "other- play" for zero-shot coordination. In International Con- ference on Machine Learning, pp. 4399-4410. PMLR, 2020.
Cover, T. and Thomas, J. Elements of Information Theory. John Wiley Sons, 1991.
de Oliveira, H. Blackwell's informativeness the- orem using diagrams. Games and Economic Behavior, 109:126-131, 2018. ISSN 0899-8256.
Hunter, J. D. Matplotlib: A 2d graphics environment. Com- puting in Science & Engineering, 9(3):90-95, 2007. doi: 10.1 10.1109/MCEE2200.55555
doi: hutp://diiong/000000....e22222.. URL scleenee/arrcccee//////999222219
Kamien, M. I., Tauman, Y., and Zamir, S. On the value of
information in a strategic conflict. Games and Economic Behavior, 2(2): 1990. ISSN 0899-8256.
Desai, N. Uncertain reward-transition MDPs for negotiable reinforcement learning. Technical report, Technical re- port, UC Berkeley, Berkeley, California, USA, 2017.
doi:
URL httpa//eerriicceee////////991919
ftttps://wwwsscieeeee
Fern, A., Natarajan, S., Judah, K., and Tadepalli, P. A Docision-theoretic model of assistance. JAIR, 50(1):71- 104, 2014.
Koller, D. and Friedman, N. Probabilistic Graphical Mod- els: Principles and Techniques. MIT Press, 2009.
Kuhn, H. W. Extensive games and the problem of in- formation. In Kuhn, H. W. and Tucker, A. W. (eds.), Contributions to the Theory of Games (AM-28), Vol- une II, chapter 11, pp. 193-216. Princeton Univer- sity Press, Princeton, 1953. ISBN 978140083 1970. doi:
Forges, F. An approach to communication equilibria. Econo- metrica: Journal of the Econometric Society, pp. 1375- 1385, 1986.
Goldman, S. and Kearns, M. On the complexity of teaching. Journal of Computer and System Sciences, 50(1):20-31, 1995.
0001999777711100089097-0000000 URL https: / //doi. 1ww..111111111011111
Laidlaw, C. and Dragan, A. The Boltzmann policy dis- tribution: Accounting for systematic suboptimality in human models. In International Conference on Learning Representations, 2021.
Google. Try Deep Research and our new ex- perimental model in Gemini, your AI assistant,
2024. URL //ttos://blo..ee http:////bbooggooggggg//otteee gemini/gpooree-gemi-ii
Lang, L., Foote, D., Russell, S., Dragan, A., Jenner, E., and Emmons, S. When your Al deceives you: Challenges with partial observability of human evaluators in reward learning. arXiv preprini urXiv:2402.17747 2024.
Google. Long context, 2025.
URL https:
wel.gollll..eee.. nn-aniiiip/oooo /occs/ Long-context
Hadfield-Menell, D., Russell, $. J., Abbeel, P., and Dragan, A. Cooperative inverse reinforcement learning. Advances in neural information processing systems, 29, 2016.
Lehrer, E., Rosenberg, D., and Shmaya, E. Signaling and mediation in games with common interests. Games and Economic Behavior, 68(2):670-682, 2010. ISSN 0899-82566 doi: http:///laaioggg//..000./.ee2..
HadfieldMMenel, D., Dragan, A., Abbeel, P., and Rus- sell, S. The owi-s game. In Proceedings of the
007. URL o.tph///.itttss///w..seeeeddedda

---

Observation Interference in Partially Observable Assistance Games
Luce, R. D. Individual Choice Behavior: A Theoretical Analysis. Dover, Mineola, NY, 1959.
Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman (eds.), Proceedings of the 9th Python in Science Conference, pp. 56 - 61, 2010. doi: 1.2250000Majorra222111111
McFadden, D. Conditional logit analysis of qualitative choice behavior. In Zarembka, P. (ed.), Frontiers of Econometrics. Academic Press, New York, 1973.
Williams, M., Carroll, M., Narang, A., Weisser, C., Murphy, B., and Dragan, A. Targeted manipulation and deception emerge when optimizing LLMs for user feedback. arXiv preprint 2:Xiv:2411.02.000 2024.
Ng, A. and Russell, S. Algorithms for inverse reinforcement learning. ICML, 2000.
OpenAI. URL
Introducing ChatGPT search, 2024. https://openali com. www.coomiini.. index/ search/.
Woodward, M., Finn, C., and Hausman, K. Learning inter- actively learn and assist. In Proceedings of the AAAI con- ference artificial intelligence, pp. 1535-25433 2020.
pandas development team, T. pandas-dewpandds: Pan- das, February 2020. URL 5281 / htt.///eoddd.3.300 3509134.
Ziebart, B., Maas, A., Bagnell, J., and Dey, A. Maximum entropy inverse reinforcement learning. In AAAI, 2008.
Park, P. S., Goldstein, S., O'Gara, A., Chen, M., and Hendrycks, D. Al deception: A survey of examples, risks, and potential solutions. Patterns, 5(5), 2024.
Ramachandran, D. and Amir, E. Bayesian inverse reinforce- ment learning. In IJCAI, 2007.
Reif, H. The complexity of two-player games of in- complete information. Journal of computer and system sciences, 29(2):274-300 1984.
Russell, S. and Norvig, P. Artificial Intelligence: A modern approach. Pearson, 3 edition, 2010.
Shah, R., Freire, P., Alex, N., Freedman, R., Krasheninnikov,
D., Chan, L., Dennis, M. D., Abbeel, P., Dragan, A., and Russell, S. Benefits of assistance over reward learning, 2020.
Treutlein, J., Demnis, M., Oesterheld, C., and Foerster, A new formalism, method and open issues for zero-shot coordination. In International Conference on Machine Learning, pp. 10413-10423. PMLR, 2021.
Ward, F. R., Toni, F., Belardinelli, F., and Everitt, T. Hon- esty is the best policy: Defining and mitigating AI de- ception. In Thirty-seventh Conference on Neural In- formation Processing Systems, 2023. URL https: //openreview.cocm wtt.:swwwwwwweeeeeeemmme forum? dd--ExpDDiPRRR
Waskom, M. L. seaborn: statistical data visualization. Jour- nal Open Source Software, 6(60):3021, 2021. doi: 10.21 10.21 00/2 URL https: http:://oooooooooo 21 21105/joss. 03021.
Wen, J., Zhong, R., Khan, A., Perez, E., Steinhardt, J., Huang, M., Bowman, R., He, H., and Feng, S. Lan- guage models leam to mislead humans via RLHF. arXiv preprint XrXiv:2409. 12822, 2024.
12

---

Observation Interference in Partially Observable Assistance Games
A. Proofs for Section 2.2
Our techniques are similar to those of Shah et al. (2020) and Desai (2017), who show how to form a single-agent POMDP for A by embedding H into the environment dynamics. However, our construction works in the opposite direction, with H embedding A's actions and observations into the environment.
Proposition 2.2. Suppose A is playing a fixed policy. If knows A's policy along with the POAG specification M, then H can form calibrated beliefs about the world state. For any timestep i and state St, H can P(st o the probability of 8t given H's observation history o
Proof. We construct a single-agent POMDP (S,A", T, T,R. O" .P. for H. Standard POMDP inference lets H form P(&e of),, which includes P(s: off.)
Consider a new set of states & E S₁ St+1 x (DA) x AA, where each new state & corresponds to full sequence of original states So:1, full sequence of assistant observations of and the previous assistant action a The new I' satisfies f(6++1 | Bt, ) (a ( st,at at (A)
8 afa). The new ÖH satisfies Ó (0 a H) = (H 8141, at a.). The new reward function R can be arbitrary, as it doesn't affect inference.
Proposition 2.3. Suppose A is updating its policy each iteration of the game. Knowledge of the game dynamics, of A 's initial policy, and of A 's update rule is sufficient for H to form calibrated beliefs about A 's future policy and of the world state.
Proof. Within each iteration of the game, H does the same as for Proposition 2.2. Between iterations, H applies A A's update rule to get A's policy for the next iteration.
Remark 2.4. Propositions 2.2 and 2.3 hold even if A is interfering with observations (Definition 3.2).
Proof. The possibility of observation interference (Definition 3.2) is merely treated like any other part of the other agent's policy and the game dynamics. By definition, interference actions are just another action, and our proofs of Proposition 2.2 and Proposition 2.3 made no assumptions on the actions.
B. Proofs and Example Formalizations for Section 4
B.1. A Lemma about Policies with Internal States
In our proof of Theorem 4.2 (and our proof of Theorem 4.7), we will construct policies that maintain an internal state (the previously sampled garbled observations). We will call this virtual state. However, our setup (in line with the norm in the literature) does not allow for such policies. We here show that any policy with virtual state can be "simulated" by policy without virtual states. Since this result is about a single player's policy, holding the opponent policy fixed, we will prove this in POMDPs.
First, a virtual-state policy a family of distributions T(a, T u. h), where:
h is history of observations and actions usual;
v is am agent state from some discrete set (c.g., IN or x A):
δ is another (new) virtual state;
a is an action.
Additionally we specify initial virtual state ui- Virtual-state policies give rise to histories in the obvious way: the initial agent state is un; the agent then samples an action ag and a following virtual state vn from s(. vo). In the next step it samples an action and agent state from T-- 00a v1) and so on.
We now show that policies with a virtual state can be transformed into behaviorally equivalent policies without an agent state.
13

---

Observation Interference in Partially Observable Assistance Games
Lemma B.1. Let a be a virtual-state policy. Then there exists regular policy the resulting distribution over (environment state, observation, action) histories is the same under and n. In particular, the expected rewards of the two are the same.
The result is related to Kuhn's (Kuhn, 1953) proof of the equivalence of behavioral and mixed strategies in perfect-recall extensive-form games.
Proof. For this proof we use h to denote observation histories and 02.0 to use state state-obserratiion-acion histories. Consider म that at time step is defined by
((A h(a)==
= Σ P(oo...., 24 x, (((A Vt, haa).
P0
Intuitively, at time step infer probability distribution over histories of virtual states and in particular uts conditioning on the observed observation-action history h, and then sample from the action distribution induced by π x<A A vr, ho.9).
We prove that for each time step f, the state-observation-aciion history until time step & is the same between and म. We prove this by natural induction. The base case trivial. Assume that the distribution over state-observation action histories up until time step is the same. We will show that for each state-ohservation- accionn history, the distribution over actions a++1 time + 1 the same under ı and æ. Note that the action distribution under is given by
Σ P(vo.... ... Ue n., ((.-99))(A Ut, h(*.).
2/g o
Now note that P(vo...., Ut| n. (r,1):0.0) = P(ro....,, dy ग, ho.9), i.e., given the history of states and observations, the environment states don't 't provide further evidence about the agent states, since every dependence between environmental states and agent states is mediated by observations and actions. Thus, this distribution the same as the distribution (AAl h..a).
B.2. Proof of Theorem 4.2
A for Theorem 4.2. Let M be any POAG. Let A have no private information. Then there an optimal policy pair (TH (TH,,AA) M in which A does not interfere with observations at the action level (and n observes naively).
Proof sketch. Note first that because our setting is common-payoff and involves no absentrindedhnss/imprreett recall, there is always optimal policy pair in which neither A nor H randomizes in any observation history. Let (, AA be any optimal policy pair for M. Let a intrrrre be an interference action played by π². Let ã be the corresponding non-interference strategy. Now consider the policy = that plays like A except that it plays ã instead of A iittererre'
We will now construct a corresponding human policy H that results in playing the same actions at each point as a Note that by the assumption that has no private observations and the fact that π and _ are deterministic, H always knows A' full observation history. Thus, H knows in particular when for which time steps in her observation history A would have played A @iterreer and A played A instead.
Now let F be the observation translation function as per Definition 3.1. Intuitively, we want # to apply F to any new observation that results from playing â^ rather than A interfere and then remember that modified observation in place of the actual observation. It would then be easy to show that & would result in the same actions as , Together with the fact that A fitterfere and A have the same effect on state transitions and rewards, we would immediately obtain that ( H n A) has the same utility as (Tr AA)
Unfortunately, if F is stochastic, the above construction requires that H can remember the results of past applications of F. That is, if at time step / she observes according to A and translates according to F to obtain some new observation of (that she would have obtained under interference), then at any time step 1' > i, she needs to remember that she sampled off from F. Our formalism doesn't allow for such memory. However, by Lemma B. we can construct a policy without internal memory to imitate the policy we constructed.
14

---

Observation Interference in Partially Observable Assistance Games
B.3. Formalization of Example 4.3 and Proof of Proposition 4.4
Example 4.3. H has typed apt list -a cuda see the list of cuda versions available be installed. Out of 10 total versions, only (non-empty) subset available. And of these available versions, only subset are compatible with the other environment software.
First, A takes an action. For each of the 10 total cuda versions, A can choose to or not to suppress it from the list of available packages. This gives A 21 total actions, where 1 action is non-observation interference (suppressing nothing), and the remaining 2 1 actions interfere with observations.
Second, H takes an action. H has 10 possible actions which try to install the corresponding version of cuda if appears in the version list. If available cuda version that is compatible with the other environment software is installed, yields I reward. Otherwise, it yields 0 reward.
Suppose A sees which versions are compatible with the other software in the environment, but H doesn't t. Then A 'ss optimal policy is to suppress the versions of cuda that are incompatible.
Formalization:
S = (10, x (J x (0, ((B is a terminal state, which we use to make the POAG effectively episodic. I is an initial state. The first bit, which we denote by 8o, encodes the time step. The next ten bits encode which versions are available. The last ten bits encode which versions are compatible. For any state 8, we use 8p refer to the first entry of the state.
S = {o, 1)100 [null} - representing the availability bits.
QA = {0. 11 [null) representing which packages are compatible.
θ {0) is a singleton.
AH {1,..., 10} representing which package to choose.
AA (0, 1) 10 representing for what packages, availability is suppressed, where 0 indicates suppression.
A's 's observations are given as follows. I & {E, 7} and 8p (i.e., it is the first time step), then O (o |s, a .aH) - That is, A observes perfectly what cuda versions are compatible. Otherwise, O4 (o (s, a ,at). 1|o^=null] That is, in all other time steps, A does not observe anything.
H's observations are given as follows. If {E,4) or so # then H simply observes null. If {E,1} and so then (o (s, a H) 1|0 = That is, for each availability bit, H observes 0 if A set the availability bit to 0; otherwise, H simply observes the availability bit.
R(s, H A 0 s [) so 0. Otherwise, R(s, H a²) Sa That is, reward of 1 is obtained if and only if the cuda version chosen by H is both available and compatible.
Po(s) 1[s IJ. That is, the initial state is always
If T, then T(. 8, H A is the uniform distribution over states s' in which at least one cuda version is available and compatible, i.e., E 8;81410 2 1. If s + I, then T(s') | ss a H aA if
50 0, 5/ 1 and 31:20 $1.20'
- 8n and = Fc; or s s' E.
Otherwise, T(s' 8 H A 0.
Propusition 4.4. There exists a POAG M where all optimal policy pairs (T π have that A interferes with observations at the action level and that H observes and acts naively.
15

---

Observation Interference in Partially Observable Assistance Games
Proof. Consider Example 4.3.
First consider the following policy pair: At the first time step, A chooses A E {0, 1) i... A chooses to suppress the availability signal exactly for those cuda versions that aren't compatible. At all other time steps the assistant chooses uniformly at random. Call this policy #^
At the second time step, when the human observes o E {0, 1) the human chooses some H s.t. of = That is, H chooses a cuda version that her observation shows available. It easy to see that under the above policy there always exists such a H At all other time steps, H chooses uniformly at random. Call this policy a
It's easy that the above policy pair is optimal: By the structure of the environment, most by having the human choose compatible and available policy at time step 1. Clearly, the above policy achieves this reward of 1.
receive reward of
Next, note that the only non-inter non-interference action for A (1, 1..,, 1). Thus, the only non-interference policy for is to always play (1, .,., 1). Call this policy "
Note that the best response for H against mh is 7 Thus, H is acting naively.
Furthermore, note that #H acts naively.
It is easy to see that adding a H + A communication channel makes no difference to the above analysis.
B.4. Proof of Theorem 4.5
Theorem 4.5. Let M be any POAG, and provide A with unbounded communication channel to H, forming M MA-H Then there is an optimal policy pair (x* x- for MA-H where π does not interfere with observations at the action level and H observes naively.
Proof sketch. Roughly, take any deterministic optimal policy pair (T ^^) Consider the assistant policy T that at each time step communicates A 's full observation to H and that replaces interference with non-interference actions. Because x deterministic, H can infer what A would have communicated based on A communications. The rest the proof goes the same way as Theorem 4.2.
B.5. Proof Theorem 4.7
For the proof of Theorem 4.7, we'll use the concept of entropy. For any probability distribution over some discrete space, let (P) = =-E, Pxx) log P(x) denote the distribution's entropy. The following is well-known result in information theory [e.g.. 3, Theorem 1.4.5; 10, Theorem 2.6.5).
Lemma B.2 (Conditioning decreases entropy). Let X,Y be random variables, then Ey [P Y)) s H(P())) Further, the inequality is strict if X and Y are not independent, i.e., if P(X) # P(X | v) for some y, then Ey [PX Y)) < H((()))
Using this result, we can provide the following variant.
Lemma B.3. Let S be random variable. Let X,Y Y be independent samples from F(S) and let Z be sampled from G(Y). where F and G are stochastic functions. Then
Ez (H(PSS) 2)M 2 Ex [H(((S) X))1 [X)P
Moreover, the inequality is strict if and are dependent given Z.
Proof. For the non-strict version:
HPP(S) X))
H(P(S Y))
H(P(S Y,Z))
H(P(S) Z))
Lemaa B.2
The strict version can proved the same way using the strict version of Lemma B.2.
16

---

Observation Interference in Partially Observable Assistance Games
Next, we can use this to prove that garbling induces a lower-entropy distribution over states.
Lemma B.A. Let L be some set of states. Let (Pa(- s)) and (Pol. 8))BEL l be families of probability distributions s.t. Pa is strictly more informative than P. with transformation function F. Further let S be some random variable over with full support. Let Xa
Pa(- S) and Xs F(X.). Then S and X are dependent given Xg. In particular, from Lemma B.3 get that Ex (P(S X)) < Ex H(P(S) [ X
Proof. We prove the following contrapositive: if Xa and S are independent given Xs. then Po is at least as informative as Pa. If Xo and S are independent given Xo, then we have that P(X. S) P(X6 X.). Thus, for all states we have that
P(Xa 1s)
P(as P(ab |s)
xb. 8)
Σ P(ab P(X mb)-
But this means that if we sample X3, according to Pb. and sample Xa according to P(Xa æ5), then we obtain sample for Xa according to the distribution P(Xa| |8) (i.e., Pa). Thus, we have that Po is at least as informative as Pa.
Theorem 4.7. Let M be any POAG. Then there exists an optimal policy pair (= a ) for M s.t. xA does not interfere with observations at the policy level.
A Proof. We will explicitly choose a policy for each time step t = 0, 1, .... So let's take Tho-11 1 Tor- 1 as given. Now let II, be the set of policies at time that are part of policy pair (mt, mT. that is optimal holding fixed moa-1: -0:-1- Note that the expected utility of policy pairs a POMDP is continuous. It follows that II is closed (i.e., that every convergent sequence of policies II, converges to a policy in II.).
Now from I1, choose म.d as the minimizer of
E EOH
[H(P(S++1 1OH 7 randomm 10:t-1 n
Tanddmm: "Dat- x
where H denotes Shannon entropy and n Trandom is the human strategy that chooses uniformly at random. (Note that the above entropy function is not the only function we could use for this proof.) That is, let o. be the policy that minimizes the entropy of H's probability distribution over world state. Because the given function is continuous and II, is closed (and bounded), this minimum exists (by the extreme value theorem).
Now by Lemma B.4 we have that if o.A more informative than A then Te will also have lower entropy at time follows that there is no policy in I1, that is more informative than म/A
Finally, it is left to show that there is policy Te outside of II, that is more informative than nं. For this, the same argument in the proof of Theorem 4.2: if there were a more informative म.f with the same effect on state transitions, then this would also be part of an optimal policy pair (constructed by having H apply the appropriate garbling internally). But we have already that in II, there is no more informative policy than A.A
Note that the mntropy-minimizing policy used in the proof may still interfere with observations at the action level. For example, by default H might receive a low-information signal about the world. The suiropy-minimizing policy might be one which A overwrites this default signal in a way that expresses more information about the world. For instance, let's assume that by default, H observes a random number between -20 and it's cold outside and a random number between 0 and +40 if it's warm outside. A receives various hints about the temperature and can overwrite the signal with arbitrary number. (Le., for each number between -20 and +440, there's action that sets H's observation to be that number.) Assuming nothing else happens in this POAG, the entropy-minimizing policies will be ones that overwrite the signal in a way that encodes 'ss information about the temperature. For instance, A it may (or may not) be an mon-interfering-att sattringgtttthe-ppee the-pooicy-leve strategy for A to overwrite H's signal with A's expectation of the temperature in degrees Celsius. Given such a policy, the entropy of H's beliefs about the world is lower than before (H has more information about the temperature). But each of these overwriting actions individually iibsevatiooninntrferfi
17

---

Observation Interference in Partially Observable Assistance Games
C. Formalization of Example 5.1 and Proof of Proposition 5.2
Recall the example:
Example 5.1. H would like to schedule a job on a cluster. She can choose between two nodes. By default, she receives a from the environment about the two nodes' specifications. Each node may be either GPU-optimized or CPU-optimized. Also, the CPUs may be either AMD or Intel.
H has strong preference between GPU-optimized and CPU-optinized nodes. She has weak preference benween AMD and Intel. These preferences are unknown to A.
A can interfere with H's observation about the available nodes. In particular, A can make it so that a choice between two CPU-optimized nodes appears as a choice between a GPU-optimized and CPU-optimized node. A observes H's choice. Later, A is charged with scheduling a job for H and has to choose between a CPU- and a GPU-optimized node on H's behalf.
If H chooses naively upon seeing only CPU-oplimized nodes (simply choosing her favorite), then A's best response interferes with observations at both the action and policy levels. Interfering with observations allows A to learn H's preference about GPU- vs CPU-optimized nodes.
In particular, there are four possible states: (1) The first node is GPU-optimized and the second node is CPU-optimized. (2) The first node is CPU-optimized and the second node is GPU-optimized. (3) Both nodes are CPU-optimized, The first has an Intel processor, the second has an AMD processor. (4) Both nodes are CPU-optimized. The first has an AMD processor and the second has an Intel processor.
Suppose the utilities of the human choice are given as follows: for the favored CPU-optimized type: for a GPU-optimized node if H favors the GPU-optimized node. The reward is 0 otherwise. On the second step, the reward for the favored type of node is 10 and O for the other type of node.
Recall the proposition was as follows.
Proposition 5.2. There is a POAG M with the following properties. For every optimal policy pair (T ())),,))))))))>>>)) at least one of the following holds: (i) n. is not acting naively, or (ii) T interferes with observations at both the action and policy levels. Additionally, there exists an optimal policy pair (x H A) where TH acts naively and n interferes with observations at both the action and policy levels.
These properties continue to hold if we require that in M, A has no private information or can arbitrarily send messages to H (i.e., there is POAG M s.t. M = MA-HH
Proof sketch. Consider the example. First let's consider naive human policy, i.e., one that chooses the favorite node type in the first time step. Then the best response for A is to interfere.
It is easy to see that in all optimal policy pairs, A must learn about H's GPU-versus-CPU preference. It follows that at time step 1, H must deterministically choose depending on her GPU-versus -PU preference.
easy to see that all of these policy profiles have the same expected reward as the above naive/interference policy pair.
Note that in the above example, A has no private information. It is easy see that the above argument continues to go through if we allow A to send signals to H.
D. Formalization of Example 6.2 and Proof of Proposition 6.3
Definition 6.1. Let M be a POAG. Let A be A's policy in M. We say that H's policy T is a Doatzmann--aations response to A if there exists some 3 > 0 s. for every human observation history h that arises with positive probability in M under
(nAA x H) we have that H (a| h) x exp (SIE -E==nd R(St, AA,AH) "H,,,AA,, nn.
Example 6.2. H is running a terminal command and is unsure whether to run the command with flag I or flag 2. With equal probability, either flag 1 or flag 2 is better, and how good the flags are differs by either a little or a lot. The worse flag always yields a utility of 0, while the better flag either yields a utility of I or a utility 7. Thus, H is uniformly at random in one of four states. A has two actions: man and tidr. The man page is a long docunent that tells the human exactly what the values of the flags are (i.e., the exact state: which flag is better and whether its utility is or 7). The t 1 dr page is a
18

---

Observation Interference in Partially Observable Assistance Games
short summary that tells the human which flag is better, but not by how much (i.e., ruling out half the states, leaving half remaining). Thus, the expected wfility of the better flag 4 (and of the worse flag is 0).
With uniform probability, H is in one of four possible states:
Flag is better by a lot: flag has value +7, while flag 2 has value O.
Flag is better by littl: flag has value +1, while flag 2 has value O.
Flag 2 is better by little: flag has value 0, while flag has value +1.
Flag 2 is better by lot: flag I has value 0 while flag 2 has value +7.
This gives us the following formalization for the game:
S ({0,1} {sa: Sb, Se, sa)) U {}
QH = (1,2) U {mill}
Q S2H
θ is singleton
A
{1,2)
A (tidr,man}
H's observations are given as follows. For s {Sas Sb. Se, sa). we have (H || (0,s), a 1 H
s). and
for E {1,2} we have Oi (0, s), tldr, at) 1li []1[s e {sa, Uh 1li 2]1[s E (sc, Otherwise,
H's observation is deterministically mull.
A's 's observations the same as H's observations.
The reward is given as follows:
R((1, 1, 1, =7
(1)
(b(11,8),, 1,aA)=1
(2)
R((L, 2,) =7
R((1, 2,),
(4)
1[a 1]1[s [sa, + 1[a 2]1[s (se, sa All other rewards are 0.
For all HH A T(. 1, T, A the uniform distribution over (0) x {8a; 8b, Sc, Sb; 8a). For s E (Sa, 8b, 8a) T(s' (0, s), H a4) 1|s (1) For all s, T'ss' (1, a aA) 1[s' E}. Finally, T(s' E, E, aa 1[s' E).
Proposition 6.3. For every B > 0, 3 POAG in which neither H nor A has private information s.I. all B-Boltzmann- rational/optimal policy pairs HT A have A interfere with observations at both the action and policy levels.
Proof. Note first that multiplying 3 by any positive number has the same effect on Boltzmann-ratione strategies as multiplying all rewards by that number. Therefore, we can consider 3 1 without loss of generality.
Consider Example 6.2. Note that tldr an observation interference action man results in a more informative signal to H.
Now consider the non-interference policy for A that always plays man. Then Boltzmann-rational H will choose follows: If she observes then she will choose an expected utility of 7 with probability x exp(7) and an expected utility of 0 with probability x exp(0). Thus, the expected utility is
exp(1
(5)
exp(7) + (x)
19

---

Observation Interference in Partially Observable Assistance Games
expected utility without
minus with interference
0.3
0.2
0.1
B
0.5
1.0
1.5
2.0
2.5
-0.1
Figure 2: The effect of varying B on the assistant's incentive for observation interference in Example 6.2. Specifically, the y axis indicates the difference between the expected utility under non-interference minus the expected utility under interference.
Similarly, if she observes 8 her expected utility is
exp(1) exp(1) (x)
(6)
Thus, overall her expected utility is
1 exp(1) exp(7) + exp(0)
exp(7)
+
2 cxp(1) + cxp(D) Ri 3.86234.
(7)
Now consider the interference policy for A in which A always plays tldr. Then upon observing either 0 or 1, the human chooses between utility of 0 and utility of 4. Thus, the expected utility is
exp(4) 2 3.92806.
4 exp(4) + exp(O)
(8)
We observe that this expected value under interference is higher than the expected value under non-interference.
E. Effects of Varying the Boltzmann Rationality Parameter (3)on the Assistant's Incentives to Interfere with Observations
As noted in the main text, in Example 6.2, we have that for low values of the rationality parameter 3, A prefers non- interference, while for large values 3, prefers interference. Below we will show that general, counterintuttively. A prefers non-interference for sufficiently small (positive) values of (3.
We here only consider the case of single decision. Consider case with actions. Let the expected utilities of the different actions without information be yu,1, .... Vo.n Now imagine that H might receive k different signals with probabilities P1. ..., Py- Under signal i [1,...kk k), the expected utilities of the different actions become Vi,1, y yf By the tower rule we must have for each action E {1....,n) n.,
P 30.a.
(9)
20

---

Observation Interference in Partially Observable Assistance Games
Note that without further restriction, the above setting includes settings in which the signal provides information what action is best.
For any B, the expected utility without the signal is
exp(3yo.a)00..r
(10)
m a==
exp(3yo.a)
am
The expected utility with the signal is
Ps Ea=1 exp(3y),a)
exp(3ys.a)s.aa
(11)
Proposition E.1. For all (ys, E R).E(0.1.....)) (),, (E (p. [R) satisfying Equation (9), we have that for sufficiently small but positive 3, the expected utility without the signal is most the expected utility with the signal.
Proof. It's easy to see that for B 0, the two expected utilities are the same. Thus, all we need to show is that the derivative w.r.t. 3 of the term in Eq. 11 at 3 = 0 exceeds the corresponding derivative of the term in Eq. 10.
The derivative w.r.t 3 at = of the term Equation (10) is
V0,a
Σ
y0
(12)
n
m
Note that this exactly the variance a random variable that is uniform over a= Similarly, the derivative of the term in Equation (11) is
11
1
Ps
Us
Σ
Us.a
(13)
n2
n2
Note that this is the weighted average (over s) of the uniform random variables over (Js.)a=1......
We can now prove the claimed inequality using the convexity of the square function, Equation (9) and some basic term
21

---

Observation Interference in Partially Observable Assistance Games
manipulation.
2
y0.a
2
Yu,a
(14)
π2
yo,a
2
Σ
yo,aa
(15)
n
y0,a
y0,al
(16)
o'=1
k
&
2
Σ
Σ
Payaa
Paylaaa
(17)
Equation (9)
a=l
a' =1=10 8m
k
2
ys,a
Ms.a'
(18)
aml
a' =1
2
n
1
11
Ps
ys.a
ys.a!
(19)
is convex
T1.
a'=1
2
Ps
Usaa
1
Σ
Us.a
(20)
n
n.
a'=1
2
Σ
Ps
Vara
Us,a
(21)
n
We have skipped over some term manipulations in Equations (16) and (21), both of which are essentially the equality of two definitions of the variance: Var(X) (X - (X)) and Var(X) = E[X²] E
It's interesting note that this is essentially the proof that the variance (over a) of the expectation (over 8) is at least the expectation (over 8) of the variance (over a).
Second, we want to show that for large B, A prefers observation interference, i.e., prefers to have the human choose based on the expected utilities yo.1, yo,n rather than the expected utilities that arise from further signals. However, for this to hold we need a further condition. Note that the general formalism above, the signal s may provide information about which action is best. If this is the case, then it is easy to show that for large enough 3, A will prefer providing the signal. However, consider specifically those cases in which the signal only provides information about how much better the best action is compared to other actions. Therefore, require in the following result that the best action is the same (WLOG 1) across
Proposition E.2. Let (ys,a E () (Ps E R)aEs satisfy Equation (9) and let yo.u > Ve,o for all s E (0) a E u......). Then for all sufficiently large 3 we have that the expected utility without the signal is at most the expected utility with the signal. The inequality is strict if the signal is non-trivial (i.e., ys is not constant across for some a).
We first provide a very rough sketch. For simplicity, let's say that the signal provides evidence about how much better the first action is compared to the second-best action. Then sometimes the signal will decrease the difference in expected utility between the best and second-best utility. We will show that as - co, the overall effect of learning the information is dominated by taking the best action less in this case.
We will use the following lemmas.
Lemma E.3. Let the differences between the top k actions be constant across signals and let the difference to the l + 1-th action be non-constant. Then there is a signal 8 s.t. the difference to the + 1-th action decreases under that signal.
Proof. Let k- 1 be the k-th best action according to 0 and let k be the is + 1-th best action according to 0. By the tower rule (Eq. 9), V0,A-1 - Vo.k must be greater than Us,k-1 - Ys for some 8. (If the difference in these expected utilities changes
22

---

Observation Interference in Partially Observable Assistance Games
when the signal is observed, then it must sometimes decrease.) But then in cases where this difference decreases observed, we clearly have that the difference between of the best actions to the k 1-th best action under also decreases.
Proof of Proposition E.2. The gain from obtaining the signal is:
exp( Eas exp( (3yp,a')
exp(By0o.o) exp( (3xp(330..r))
) Ma,a-
Σ Pa Σ
WLOG let O be the best action under all signals, the second-bes, best and so on. Let & be the largest number that the differences
between the utilities actions ...., - 1 are always the same. (Typically k = 0.) Let S be the set of signals under which the difference to the utility of (the l + 1-th best action) is minimized. Note that particular, the difference must be smaller than under 0 by Lemma E.3. WLOG assume that for all signals, k among the k 1-th best actions.
WLOG assume that ys,a 0 for all {0} U S and all and that ys,0 is constant across
Now we will divide up the above sum into three components:
A The change (decrease) in utility from playing the top actions less in S than without the signal.
exp(Bys,a)
exp(Byo,a)
A
Σ Ps ãc
Σ
Ene exp(
>n a' exp(33000a)
ya
a=o,...kk=1
B The change in utility from the changes in distribution of all actions other than the top i under $ versus S
exp(Byaaa)
exp(Byo..) a' (xp(3yy,a.)
B
Σ
Σ
Vä.a
exp( (xy.(
BE
k+k,k
The change in utility from all signals other than S, i.e.
exp(Bye..)
exp(BByo.o)
ΣP Ps Σ 25
En (xpp(3ye.')
En. exp(Byo,..)
ys.a
We will show that the effect from A (which is negative) is becomes infinitely much larger than the effect from B and C (in absolute terms). From that it will follow that the original sum, which is equal to A + B C' is negative as B cc.
We first provide a bound on A. We first show that A < O. To show this, note first that in all enumerators in A, we can replace Vs. with yo,a (by choice of & and kt). So all we need to show is that the second denominator is smaller than the first, ie.. E.' exp(3ya.. > E., exp(Byo.a) But this this is easy to see from the fact that Vs = yo.a for a = 0, 1..., k and ya,k > yo For large B, (3p()) will be much larger than Ea Ea'=kktl... exp(3yo.x)).
23

---

Observation Interference in Partially Observable Assistance Games Next, we will provide a lower bound on the absolute value of |A].
exp(3ys,a)
expl (Sp(()y,a)
A
Σ
Pa
En expl
exp((xxxx)a (3yaar)
End (xp(3)),a')
Všaa
aES
a=0...,k-1
En exp(330,00) ) yo,a
1
Σ $ES
Ps
exp(3yu,a)
E exp(3yaa.o)
=
Σ
Ps
exp(3yo..a
a' (xpp(yys.a) Ea. exp(3yy,.i
1
y0.a
SES
a
a= k-1
exp(33yo.a))) Ene exp(3ys,al
= Σ
pj
exp(33o.a)
En exp(Byy,, ( (Ear exp($1yo.a)) V0.0
RES
a=D....k-1
Ea=kittI... exp(3yo,ar)) Ea=k,htI... exp(Bys,o) Ca exp ( (E exp(By0..r)) y0,a
=
Σ
Pa
exp(3yo.a)
iES
a=0,...k-11 k-1
n (op(33yo.) exp(( (Bys,))
Σ
P3
exp(By0..)
n² exp(3yyo.a)
10,aa
SES
a
1=...kkk11
exxp(8
exp(Bys.k)
Σ
20.0
n2 eexe( (3y00))
a=
exp()ya,))
<
Ps
2 eexp(Byo... V0,a
2
SES
expl 3yz,k)
<
2 exxp(3y..) 30.0
2
$ES
Next we upper bound B. First, the best case for the effect on is that all the probability mass that under 0 is on the top A: actions ends up on the k-th best action, i.e.,
exp( exp(3yo..) (3yo,a)
B
ps
Σ
Ear exp((xx0..l
ya.k.
S
o=0....k-1
We can further upper-bound this as follows:
exp(3y0,o)
Σ
ps 1
Σ
E.. exp( exp((yyo.r)) 2a.
se
a=i....k-
exp(
Σ SE$
ps
exp(3yo.a.)
nexp(Byo...
Σ
Ps
exp(3yo.o)
ya.k
BEŠ
From the fact that Vs.k > Yo.k. it is easy to see that this term vanishes in absolute value relative to our upper bound on A.
Finally, we must upper bound C. First, we can upper bound C by considering a case where all probability mass that in 0 was outside the top l actions, goes to the best action when a signal outside of S is observed, i.c.,
exp( (3yn,a)
C ΣEe P
Σ
E. exp(3yyu,r) yo.0-
$55
k=k.k+1...
24

---

Observation Interference in Partially Observable Assistance Games
We can further upper bound this as follows:
exp (30,,))
Σ Ps #48
Σ
En exp()3y..p Vo.0
a=k,k+1,..
Σ
n.
Ps
exp(330..)
10,00
S
Again, from the fact that Va.k > yo it is easy to see that this term vanishes in absolute value relative to our upper bound on A.
F. Proof of A's Best Response in the Product Selection Game
Proposition 7.2. Consider A policies that always interfere with & observations for some fixed k. Among the k-interference policies for a given ki A's best response to H's straightforward product selection policy is as follows. A interferes with the k smallest RS values where Rt = R2 if A observes Ri, and R2 = 0.5 otherwise.
Proof. Consider A's perspective. A's 's interference is equivalent to selecting a set of d - k untampered products from which H selects according to a Boltzmann distribution on H;. As neither sees nor affects the H, by symmetry, over all draws of the game, selects each of the
products with equal probability. A expected payoff for choosing d products, then, is the uniform average of the products' expected U..
How does A choose the set of d k products to maximize the uniform average of the products' expected U.? Recall U, = He + R2. As A neither
nor affects the H;, A can ignore the H, and consider only the R, Denote the expected R2 by R. = E|R:]. If A observes R, then Rt; = Rg If A doesn't t observe Rú, then RG = 0.5. To choose the maximum d - k values for R,, A interferes with the minimum & values of R.
G. Minor Deficiencies of the Observation Interference Definition
As noted in the main text, there are various possible concerns with Definition 3.2 that we consider minor because they do not change the main ideas and results of this paper.
The definition does not take into account what A knows about what H already knows. As such, it will sometimes spuriously judge policy to be observation interference for taking away signal from the human that is redundant with the human's past observations. For example, if the human observes the Linux version at time / and the Linux is known not to change, then preventing the human from observing the Linux version again at time f + 1 might count as observation interference.
The definition may also spuriously judge a policy to not be observation interference because the only more informative policies fail to provide some redundant piece of information to the human. For instance, let's 'ss say that by default the human learns some new, useful information at time + 1. Now let's say that A can make it so that H instead observes the Linux version (which H already knows). Assume that has no way of letting H see both the Linux version and the new, useful information. Then making the human observe the Linux would not count sensor interference according to our definition, because definition doesn't 't take into account that the human already knows the Linux version.
Adapting the definition fix this deficiency is somewhat cumbersome, because it requires us reason about A beliefs about H's observation histories/beliefs.
This aspect of the definition seems mostly irrelevant for our results. For instance, none of our examples of observation interference have redundant observations. Therefore, we have opted keep the definition simple in this paper.
Our definition only compares pure actions in terms of their informativeness. But it may be the case that one action A is. in some intuitive sense, interferring with H's observations but the only way to show this is to compare à with a mix of actions, say, mixing uniformly over a and ag In particular, may be that has the same effect on state transitions as mixing uniformly over at and az while reducing the informativeness of the H's observation. It's easy to extend the definition to also consider mixed actions, but the extension has no impact on any of our results.
25

---

Observation Interference in Partially Observable Assistance Games
Neither the action-level the policy-level notion of tampering is sensitive what policy H plays even what policy H might plausibly play. For instance, let's say there is some action dilly for H that it never makes sense for H to play. (In game- cume-theoretic terms, it might be strictly dominated.) Then whether any given policy πA tampering will sensitive to what happens assume some degree rationality on behalf of H.
H plays π and H plays silly. Arguably this shouldn't 't matter; arguably we should
To refine this definition, we would need restrict attention to specific policies or actions for H. It's clear which restriction makes most sense. In any case, we cannot imagine refinement of the definition that would have little impact on results.
H. Code Assets
Our experiments use the Python software libraries Matplotlib (Hunter, 2007), NumPy (Harris et al., 2020), pandas (pandas development team, 2020; Wes McKinney, 2010), and seaborn (Waskom, 2021).
26


Reviews:
[{"review_id": "tDgenusOJK", "reviewer": "Reviewer_MYkQ", "summary": "This paper investigates observation interference by AI assistants in partially observable assistance games (POAGs), where both the AI and the human have limited information. The authors demonstrate that an optimal assistant may have incentives to interfere with observations to communicate private information, query human preferences when the human acts naively, and help the human make better decisions when the human is irrational. Defining observation interference as providing less informative signal, they show that while action-level interference may sometimes be necessary, policy-level interference is never required. This finding suggests that although observation interference involves sacrificing some information, it can benefit the human by facilitating the communication of more critical information. Experiments further explore the trade-offs influenced by the amount of the AI's private information and the degree of the human's rationality.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Thanks + response\n\nThank you for your review. We are glad you consider our work to \"address a crucial aspect of the human-AI value alignment problem\" and \"have important implications for building trustworthy AI systems.\" We appreciate the discussion points you raised.\n\n## 1.\n\n> The analysis… relies on specific assumptions… [such as] the Boltzmann rational model.\n\nIn this first paper on observation interference in human-AI interaction, we seek to lay a theoretical foundation using simple, well-known human models. Our results consider several models, including optimal play, decisions based on immediate outcomes, and Boltzmann rationality. Many published papers consider only the Boltzmann model, which is standard in the literature. For example, Hong Jun Jeon et al. (NeurIPS 2020) considers 11 feedback types that all assume Boltzmann rationality. Similarly, Ramachandran and Amir (IJCAI 2007) and Laidlaw and Dragan (ICLR 2022) exclusively study this model. We hope our paper will be judged by a consistent standard with other published work.\n\n**Action 1.1: In \"Limitations and Future Work,\" we will add discussion of human modeling assumptions, suggesting additional models as a direction for future work.**\n\n## 2. \n\n> the communication channels appear to assume that the human and the AI agent can perfectly understand each other\n\nWe agree that communication is not a fully general solution to observation interference. But we don't view this as a weakness. If anything, our nuanced study would be less relevant if communication was a fully satisfying solution.\n\n**Action 2.1: We will add the following paragraph on practical limitations of unbounded communication channels after Theorem 4.5:**\n\n\"One could argue that in practice, an unrestricted communication channel between A and H could usually be made available. However, Theorem 4.5 ignores various real-world obstacles. For one, it considers communication that incurs no cost, but realistically communication costs the human time and effort. Second, the optimal policy pair requires A to send information in a way that H can reliably understand and act on. We should expect that in practice, A and H sometimes cannot understand each other. Therefore, despite Theorem 4.5, we think observation interference is of broad practical relevance, even where A can, e.g., send text messages to H.\"\n\n## 3.\n\n> the paper would benefit from empirical validation with human subjects\n\nWe agree this would be valuable for future work, but we also believe that would be a different kind of paper. This first paper on the topic is about laying theoretical foundations.\n\n**Action 3.1: In \"Limitations and Future Work,\" we will discuss empirical validation with human subjects as a valuable future direction.**\n\n## 4.\n\n> Consider the computational complexity (NEXP-hard) of finding optimal policies in POAGs, how can the theoretical insights from this paper be practically applied to the real world?\n\n**Action 4.1: We will add the following discussion to our paper:**\n\n\"How might our results apply given that finding optimal policies in POAGs is NEXP-hard?\n\nMost of our paper is descriptive, characterizing when observation tampering could happen. Complexity considerations could affect these results in either direction. It's easy to construct environments where finding good observation-interfering policies is computationally intractable but constructing good non-interfering policies is easy; and vice versa. In practice, complexities of the environment can be orthogonal to incentives to interfere. For instance, a real-world version of the CUDA example is complex (A assesses complicated software compatibility issues), but the decision whether to interfere with observations is easy. We believe our characterizations remain useful even in complex environments (where we can't expect optimal policies), although we can't make as definitive claims as we can about optimal policies.\n\nWe have discussed allowing communication between H and A. A complexity-theoretic argument favors this solution: If H and A share all private information, the game effectively turns from a DecPOMDP into a POMDP. Solving POMDPs is PSPACE-complete and thus likely easier than solving DecPOMDPs.\"\n\n## 5.\n\n> the ethical implications of designing AI systems that may optimally interfere… What constraints might be appropriate?\n\nPlease see Action 1.1 in our response to Reviewer fCAi, where we discuss implications. Our theory reveals that optimal assistants must sometimes interfere with observations, so interference is a nuanced issue. Appropriate constraints might include transparency about when interference occurs, alignment with user goals, and user controls to override interference when desired. As you suggest, future work with human subjects is needed to refine these constraints in practice.\n\n## What do you think?\n\nThank you again. **Do the above actions address your concerns?** If not, what further changes could we make to improve your score?", "claims_and_evidence": "Please refer to Strengths And Weaknesses.", "methods_and_evaluation_criteria": "Please refer to Strengths And Weaknesses.", "theoretical_claims": "Please refer to Strengths And Weaknesses.", "experimental_designs_or_analyses": "Please refer to Strengths And Weaknesses.", "relation_to_broader_scientific_literature": "Please refer to Strengths And Weaknesses.", "essential_references_not_discussed": "Please refer to Strengths And Weaknesses.", "other_strengths_and_weaknesses": "**[Strengths]**\n- This paper is well-organized and easy to follow, presenting formal definitions, solid proofs, examples, and experimental results.\n\n- This paper addresses a crucial aspect of the human-AI value alignment problem by considering the more realistic scenario of partial observability. The theoretical analysis of observation interference and the three distinct incentives may have important implications for building trustworthy AI systems.\n\n- Although the authors acknowledge some minor limitations in the definition of observation interference in Appendix G, the definitions, theoretical claims, and preliminary results appear mathematically sound.\n\n**[Weaknesses]**\n- The analysis of different incentives for observation interference often relies on specific assumptions about human behavior, such as naivety or adherence to the Boltzmann rationality model. The extent to which these assumptions accurately capture real human behavior across different contexts could affect the practical relevance of the findings. Additionally, the communication channels appear to assume that the human and the AI agent can perfectly understand each other, which may not hold in more complex scenarios.\n\n- While the experimental model provides valuable insights, the paper would benefit from empirical validation with human subjects to test its theoretical predictions.\n\n- The gap between theoretical optimality and practical implementation raises questions about the design of AI systems. A more in-depth discussion of how these insights could direct the development of AI assistants may enhance the paper.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- Consider the computational complexity (NEXP-hard) of finding optimal policies in POAGs, what are your initial thoughts on how the theoretical insights from this paper could be practically applied to the design and implementation of real-world AI assistants, particularly those operating in complex and partially observable environments?\n\n- Have you considered the ethical implications of designing AI systems that may optimally interfere with observations? What constraints might be appropriate to ensure that such interference does not lead to problematic patterns of interaction?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Oue4ItHpX4", "reviewer": "Reviewer_fCAi", "summary": "Paper studies the conditions under which an agent has an incentive to perform observation interference (take an action which returns partial state observation to the human) even when the goals are aligned. The thought of taking such an action, at surface level seems adversarial and counter intuitive. However, the paper discusses conditions such as when the human is making decisions based on immediate reward, when agent wants to reveal some private information, or when the human is irrational and restricting observation is a way of forcing them to be rational. All of these comes at a cost of destroying state information which causes the trade off.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Thanks + response\n\nThank you for your review. We are glad you consider our work “very relevant” to the broader scientific literature and that “the general reasoning discussed in the paper is helpful to build intuition for the theory.” We appreciate the questions you raise about the practical applicability of our work along with the example scenarios that you pose. \n\n## 1. \n\n> What is the practical applicability of the work? Are there domains beyond the curated examples where results of the paper are be discussed? What is the main impact of these results in such domains?\n\nThank you for raising these important questions about the practical applicability of our work. To address these questions:\n\n**Action 1.1: We will include a discussion of the practical applicability of our work in the “Impact Statement” section of our paper.** We propose the following text:\n\n“AI assistants are being developed and deployed in settings where humans can only partially observe what’s happening. For example, AI assistants including ChatGPT, Claude, and Gemini can search the web while only returning summaries to users ([OpenAI, 2024](https://openai.com/index/introducing-chatgpt-search/); [Anthropic, 2025](https://www.anthropic.com/news/web-search); [Google, 2024](https://blog.google/products/gemini/google-gemini-deep-research/)). Moreover, the sorts of AI models powering these assistants are processing increasingly long inputs. Whereas the original ChatGPT model could only process 4096 input tokens, today’s Gemini 1.5 Pro can process 2,000,000 input tokens—which is roughly 100,000 lines of code, or 16 novels of average length in English ([Google, 2025](https://ai.google.dev/gemini-api/docs/long-context)). In the future, we anticipate that AI assistants will be deployed at increasing scale, independently taking more actions on behalf of users and processing increasingly long context lengths. We thus expect that over time, humans will have less and less ability to directly observe everything that’s happening.\n\nThe goal of our work is to lay a theoretical foundation for understanding when AI assistants have an incentive to interfere with human observations. Our results create a nuanced picture, suggesting that not all observation interference is inherently bad. In practice, we expect that AI assistants will exhibit observation interference for a mix of good and bad reasons. With this theory, our goal is to help practitioners disentangle these different incentives for observation interference when they emerge in practice.”\n\n## 2. \n\n> What happens if A has multiple private information?\n\nWhile we prove all our “negative” results (results showing that observation tampering can happen) with simple examples in which both A and H only make a single observation once, all our positive results apply to any POAG, and thus allow settings in which both A and H observe multiple times!\n\n**Action 2.1: To clarify how general our setting is, we will add the following sentence to Section 2.1:**\n\n“While all the examples in this paper are quite simple, note that the POAG setup and thus all our positive results are very general, allowing both H and A to observe private information at multiple times, taking actions that influence both the state and each other’s observations, and so on.”\n\n## 3. \n\n> Are actions like, attempt to open door, for an agent trying to expose that the door is locked, a valid example for agent exposing private information, (let's say human could never have been in the room). If so, this action doesn't alter the environment but exposes private information. Therefore, in the presence of both such actions - observation interfering and signaling which would be preferred?\n\nThank you for proposing this great “locked door” example.\n\n**Action 3.1: We will add the locked door example to our paper after Definition 4.6, with this text:**\n\n“While in our examples, we will mostly consider actions that in some sense act directly on H’s observations, Definition 4.6 also considers the informational effects of physical actions. For example, if A (visibly) tries to open a door that A knows to be locked, then this reveals to H that the door is locked. Consequently, not trying to open the door (when A knows it to be locked) is an instance of observation interference in the sense of Definition 4.6. While having the same (null) effect on the state of the world, trying to open the door provides H with more information about the world.”\n\n## What do you think?\n\nThank you again for taking the time to review the paper and providing helpful feedback. **Do the above actions address your questions about the paper?** If not, what further clarification or modifications could we make to improve your score?", "claims_and_evidence": "yes, the paper is great to read.", "methods_and_evaluation_criteria": "yes.", "theoretical_claims": "yes, and the general reasoning discussed in the paper is helpful to build intuition for the theory.", "experimental_designs_or_analyses": "yes.", "relation_to_broader_scientific_literature": "yes, this is very relevant to assistant games.", "essential_references_not_discussed": "discussed.", "other_strengths_and_weaknesses": "I enjoyed reading the paper. \n\nI have a few concerns : \n\n1. What is the practical applicability of the work? Are there domains beyond the curated examples where results of the paper are be discussed? What is the main impact of these results in such domains? \n\n2. What happens if A has multiple private information? \n\n3. Are actions like, attempt to open door, for an agent trying to expose that the door is locked, a valid example for agent exposing private information, (let's say human could never have been in the room). If so, this action doesn't alter the environment but exposes private information. Therefore, in the presence of both such actions - observation interfering and signaling which would be preferred?", "other_comments_or_suggestions": "see above comments.", "questions_for_authors": "see above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dKDvUCnp5G", "reviewer": "Reviewer_SZMC", "summary": "This work studies a two player decentralised POMDP called Partially Observable Assistance Games. In this game, the authors study cases where it might be beneficial to one of the player (called assistant) to \"interfere\" with the observations of the other player (called human). They also identify situations, where such an undesirable interference should not happen (e.g. without any private information, or with a free communication channel).\n\nFor a different definition of observation interference though (at policy level vs action level), they provide a result guaranteeing the existence of interference free optimal policies.\n\nLastly, this work shows that observation interference might also be optimal in the case of irrational or 'naive' human player.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Thanks + response\n\nThank you for your review. We are glad you find our setting “of strong interest,” and we appreciate your help with the clarity and simplicity of our paper.\n\n## 1.\n\n> real difference between Definition 3.2 and Definition 4.6 is unclear… I did not understand the example after Theorem 4.7.\n\nThank you - we want to be sure we clearly communicate these important points. We will:\n\n**Action 1.1: Extend our discussion on line 290 of the example after Theorem 4.7 to more thoroughly explain how Definition 4.6 differs from Definition 3.2.** The revised discussion will read:\n\n“We now revisit Example 4.3. For observation tampering under our earlier Definition 3.2, H simply knows that A has taken the *action* to suppress some versions of cuda. However, H does not know anything about A’s *policy*. For all H knows, A’s policy could be to randomly suppress cuda versions or to always suppress the same cuda version. Thus, suppressing any version is strictly less informative for H than the list of all available versions. This is why Definition 3.2 calls suppressing versions ‘tampering at the action level.’\n\nThe key difference with Definition 4.6 is that H knows A’s policy. Suppose that A’s policy $\\pi^\\mathbf{A}$ is to suppress exactly the versions of cuda that are incompatible with the other software in the environment. Because H knows that A suppressed the incompatible cuda versions, seeing the filtered list tells H which versions of cuda are compatible! Although suppressing versions is strictly less informative under Definition 3.2 (when H doesn’t know A’s policy), suppressing versions provides H with new information under Definition 4.6 (when H knows A’s policy). Accordingly, $\\pi^\\mathbf{A}$ is interfering with observations at the action level *but not at the policy level*.”\n\n**Action 1.2: Move the discussion of the example after Theorem 4.7 (which is currently on Line 290) upward to line 280 so that it immediately follows Definition 4.6.** This way, the reader will immediately see an explanation of the intuition behind Definition 4.6 before Theorem 4.7 (which relies on Definition 4.6).\n\n**Question 1.3: Do these revisions provide clarity? If not, please let us know, and we will work to clarify further.**\n\n## 2.\n\n> What is so specific to POMDP? … POMDP[s] might be overcomplicated…. A [much simpler] normal form game, could have led to the same conclusions.\n\nWe require a framework with private information and sequential play, which normal-form games lack. Observation interference (both Definitions 3.2 and 4.6) needs A to choose first, then H to observe something dependent on A's choice.\n\nWe study causes of observation interference which need:\n* For Communication (Section 4): A observes privately → A acts → H observes → H chooses (Example 4.1)\n* For Querying (Section 5): A acts → H observes → H acts → A observes → A acts (Example 5.1)\n\nWhile extensive-form games could use fewer \"dummy\" states, they're similarly complex as DecPOMDPs, which are standard in assistance games literature (Hadfield-Menell et al., 2016; Shah et al., 2020).\n\n**Question 2.1: Does this clarify why we can’t simplify our setup?**\n\n## 3.\n\n> I would need more explanation to really think this setting could apply to RLHF.\n\n**Action 3.1: At the start of Section 4.1, we will add the following explanation:**\n\n“We can model RLHF within the POAG framework as follows:\n* The assistant’s goal in RLHF is to satisfy the human’s preferences. In a POAG, this corresponds to the shared reward function $R$ which has a parameterization $\\theta$ that only H knows.\n* In RLHF, the assistant rolls out trajectories, and the human’s picks which trajectory is preferred. A POAG can model this by letting H observe pairs of trajectories explored by A but only giving H a binary action (to choose which trajectory H prefers).\n* A’s final RLHF policy maximizes an estimate of $R$ based on a dataset of H’s preference comparisons (Lang *et al.* (2024), Proposition 4.1). In the POAG framework, A can compute this policy based on A’s observations of H’s binary actions.”\n\n## 4.\n\nFor the minor comments, we will:\n* Add to line 115 “POAGs inherit the generality of DecPOMDPs: POAGs can model games where H acts first, where A acts first, or where H and A act simultaneously.”\n* Add the full name for DecPOMDP to Line 111.\n* Add a clarification so that line 132 reads: “... Nash equilibria for the shared reward function R.”\n* Add this definition of a stochastic function: “a function mapping observations to random variables over observations”.\n* Fix the line 231 typo.\n\n> how Propositions 2.2 and 2.3 are actual propositions\n\nAppendix A needs these propositions to have names in order to refer to them and prove them. To clarify, we will add “(See Appendix A for proofs.)” to line 148.\n\n## What do you think?\n\nThank you again for your review. **Do the above actions address your concerns?** If not, what further clarifications or modifications could we make to improve your score?", "claims_and_evidence": "N/A", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "Studying situations where hiding some private information, even in collaborative games, is of strong interest in my opinion. I think this is even more general than this \"assistance game\" motivation.\n\nMy main concern is about the clarity of the provided mathematical framework. Notably, there are a lot of definitions, notations and objects at stake and it is hard to understand the real meaning of all of them.\n\nAs an example, the real difference between Definition 3.2 and Definition 4.6 is unclear to me (notably because it does not affect state transition), even after reading the illustrating example. In consequence, I don't understand why this shold lead to a significant difference in the optimal policy pairs, given by Theorem 4.7 vs Proposition 4.4.\n\nAfter looking more carefully at the appendix, this seems more due to some subtleties in the definitions than a fundamental diferrence between these definitions.\n\nProbably related is my impression that the POMDP framework might be overcomplicated with the messages that the authors want to give. Indeed, when looking at the mathematical definition of the example provided along the paper, this example corresponds to a degenerate POMDP (the stationary distribution cycles in a dummy terminal state). I thus feel that a much simpler problem formulation, e.g., in a normal form game, could have led to the same kind of conclusions. As a consequence, the messages of this paper are hard to grasp, due to an unnecessary (in my opinion) level of complexity in the setting, that requires the introduction of numerous definitions and notations.\n\nInstead, I would have preferred a much simpler setting that draws similar conclusions.\n\nAdditionally, the example given along the paper indeed seems a necessary part to grasp all the presented concepts. However, I find this specific example unclear at some points, and it didn't really help me understand in the end (I particularly did not understand the example after Theorem 4.7).\n\nLastly, I am not sure of the relevance of this setting to RLHF, and I would need more explanation to really think this setting could apply to RLHF.\n\n---------------------\n# Minor comments \n\nIt seems in the paper that the assistance first takes his action, the human observes it and then takes action based on the assistance action. I did not see it clearly stated in the paper though.\n\nDefinition 2.1: please give the full name for DecPOMDP at least once\n\nLine 132: for what utilities are they Nash Equilibria?\n\nI don't get how Propositions 2.2 and 2.3 are actual propositions. They are just probabilistic statements (no need for a formal environment here in my opinion).\n\nIn definition 3.1, what is the definition for a stochastic function?\n\nLine 231 (right): I think there is a typo, it should be \"if A can communicate with H\"", "questions_for_authors": "What is so specific to POMDP in the final conclusions drawn in this work? It seems to me that all the statements are somehow consequences of statements for stateless games (eg normal form game)\n\nSee other questions above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "363ugY0SST", "reviewer": "Reviewer_dvjv", "summary": "The paper studies POAG where human and AI assistant only have partial observations. It shows that an optimal assistant who aims to maximize human reward needs to take observation-interfering actions, defined as an action showing a subset of information to human, for 3 purposes: \n1.\tCommunicate AI’s private information\n2.\tQuery human’s private information\n3.\tHelp Boltzmann-rational humans\n\n## update after rebuttal\nMy assessment has not changed after the rebuttal from the authors -- I believe that this paper makes a useful contribution to the field.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Thanks + response\n\nThank you for your review. We are glad you think our work “provides a useful contribution to the literature” with technical details that are “sound,” “effective,” and “intuitive and easy to follow.” We appreciate the questions you ask about some of our examples and experiments.\n\n## 1.\n\n> There are some conceptual similarities of this work to off-switch games (this is not an essential reference though): Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2017). The off-switch game. In Workshops at the thirty-first aaai conference on artificial intelligence\n\nThank you for sharing this reference. We agree it shares conceptual similarities, and we will cite it.\n\n**Action 1.1: We will add a citation to Hadfield-Menell’s et al.’s “The off-switch game.”**\n\n## 2. \n\n> why didn’t the paper show an experiment for purpose 2?\n\nWhile we were limited by space constraints, we think this is an exciting direction for future work. Thus:\n\n**Action 2.1: We will discuss experiments for purpose 2 as a direction for future work in the “limitations and future work” section of our paper.**\n\n## 3. \n\n> major typo: line 232 if A can communicate with A H\n\nWe will fix this typo.\n\n> minor typo: line 104 Lehrer extend(s), line 380 utilities and 1 and 0\n\nWe believe “Lehrer extend” is correct (see [rule 20 here](https://www.cs.columbia.edu/~hgs/etc/writing-bugs.html#:~:text=Note%20that%20%22et,%22shows%22.)) and will fix the line 380 typo.\n\n> p. 6, the paragraph that starts with \"At first sight\" is a bit unclear. Why present this unlikely human strategy?\n\nWe were worried that Example 5.1 might be misconstrued as a counterexample to Theorem 4.2. We included this “At first sight” paragraph in an attempt to clarify why Example 5.1 is still consistent with Theorem 4.2.\n\n**Action 3.1: To clarify this, we will do the following:**\n* Remind the reader of the content of Theorem 4.2: “which states that non-interfering optimal policy pairs always exist”.\n* At the end of the “At first sight” paragraph, add: “The key point of Example 5.1 is therefore that---while there is _some_ optimal policy pair without observation interference---there is no plausible optimal policy pair that avoids observation interference. More specifically, we use the notion of acting naively (Definition 3.7) to express this notion of plausibility and rule out the above policy. We thus obtain the following proposition, which states that in some POAGs, if we want to play an optimal policy pair and we want H to be able to act naively, then A has to interfere with observations.”\n\n> I did not follow the logic of the values of the utilities (e.g., 1,0,4,7) on p. 7, the paragraph that starts with \"Intuitively, both the tldr and man pages allow the human to choose optimally?\n\nWe agree that the logic of the utilities for Example 6.2 should be clearer.\n\n**Action 3.2: To address this, we plan to discuss the utilities in Example 6.2 itself, rather than after it.**\n* In Example 6.2, we will spell it out in more detail as follows: “Specifically, the worse flag always yields a utility of 0. The better flag either yields a utility of 1 or a utility 7.”\n* We will also expand the parenthetical “i.e., the exact state: which flag is better and whether its utility is 1 or 7”.\n* And at the end of the example, write, “Therefore, the expected utility of the better flag is 4 (and the utility of the worse flag is 0).“\n\n> For the experiment in section 7.1, there is no shared information correct? Would the presence of shared information change anything?\n\nThat is correct: there is no shared information. The presence of shared information would not necessarily change anything. As long as A has private information that is useful to the human, then it can still have an incentive to interfere with observations for purpose 1. And as long as A can make H’s decision easier by interfering, then A can still have an incentive to interfere with observations for purpose 3. This holds true even in the presence of shared information.\n\n## What do you think?\n\nThank you again for taking the time to review the paper and providing helpful feedback! **Do the above actions provide clarity to your questions?** If not, what further clarification or modifications could we make to improve your score?", "claims_and_evidence": "Section 3 sets up POAG, section 4 supports purpose 1, section 5 supports purpose 2, section 6 supports purpose 3, section 7 runs an experiment evaluating the pros and cons of interfering for purposes 1 and 3.", "methods_and_evaluation_criteria": "The authors use a very simple but effective hypothetical example to illustrate the framework. There is no real-world data to evaluate the ideas.", "theoretical_claims": "The theoretical claims appear sound to me.", "experimental_designs_or_analyses": "The experiments in Section 7 were intuitive and easy to follow.", "relation_to_broader_scientific_literature": "The paper provides a useful contribution to the literature by reviewing the conditions in which it is rational for AI to withhold information from a human user. This runs counter to the prevailing idea that the AI should reveal all available information. This paper might lead to interesting extensions where the AI might withhold not only information about the world state but also other information such as AI explanations (as this type of information often leads people to make inferior decisions). \n\nThere are some conceptual similarities of this work to off-switch games (this is not an essential reference though):\nHadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2017). The off-switch game. In Workshops at the thirty-first aaai conference on artificial intelligence", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "See comments above", "other_comments_or_suggestions": "1) major typo: line 232 if A can communicate with A H \n2) minor typo: line 104 Lehrer extend(s), line 380 utilities and 1 and 0\n3) p. 6, the paragraph that starts with \"At first sight\" is a bit unclear. Why present this unlikely human strategy?\n4) I did not follow the logic of the values of the utilities (e.g., 1,0,4,7) on p. 7, the paragraph that starts with \"Intuitively, both the tldr and man pages allow the human to choose optimally?", "questions_for_authors": "1) why didn’t the paper show an experiment for purpose 2?\n2) For the experiment in section 7.1, there is no shared information correct? Would the presence of shared information change anything?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers agreed that the paper studies an interesting problem setting of partially observable assistance games, and that the findings could be potentially useful for human-AI alignment in real-world applications. However, the reviewers also raised several concerns and questions in their initial reviews. We want to thank the authors for their detailed responses, which helped in answering the reviewers’ key questions. The reviewers have an overall positive assessment of the paper, and there is a consensus for acceptance. The reviewers have provided detailed feedback, and we strongly encourage the authors to incorporate this feedback when preparing the final version of the paper.
Decision: Accept (poster)