Learning to Assist Humans without Inferring Rewards
Authors: Vivek Myers
Evan Ellis
Sergey Levine
Benjamin Eysenbach
Anca Dragan
Venue: NeurIPS 2024 poster (2024)
Keywords: Human-AI Collaboration
Unsupervised Reinforcement Learning
Primary area: human-AI_interaction
TL;DR: We propose a scalable algorithm for assisting humans without inferring their objectives.

Abstract: Assistive agents should make humans' lives easier. Classically, such assistance is studied through the lens of inverse reinforcement learning, where an assistive agent (e.g., a chatbot, a robot) infers a human's intention and then selects actions to help the human reach that goal. This approach requires inferring intentions, which can be difficult in high-dimensional settings. We build upon prior work that studies assistance through the lens of empowerment: an assistive agent aims to maximize the influence of the human's actions such that they exert a greater control over the environmental outcomes and can solve tasks in fewer steps. We lift the major limitation of prior work in this area—scalability to high-dimensional settings—with contrastive successor representations. We formally prove that these representations estimate a similar notion of empowerment to that studied by prior work and provide a ready-made mechanism for optimizing it. Empirically, our proposed method outperforms prior methods on synthetic benchmarks, and scales to Overcooked, a cooperative game setting. Theoretically, our work connects ideas from information theory, neuroscience, and reinforcement learning, and charts a path for representations to play a critical role in solving assistive problems. Our code is available at https://github.com/vivekmyers/empowerment_successor_representations.

Learning to Assist Humans without Inferring Rewards
Vivek Myers'
Evan Ellis!
Sergey Levine'
Benjamin Eysenbach²
Anca Dragan
UC Berkeley
Princeton University
Abstract
Assistive agents should make humans' lives easier. Classically, such assistance is studied through the lens of inverse reinforcement learning, where assistive agent (e.g.. a chatbot, a robot) infers a human's intention and then selects actions to help the human reach that goal. This approach requires inferring intentions, which can be difficult in high-dimensional settings. We build upon prior work that studies assistance through the lens of empowerment: an assistive agent aims to maximize the influence of the human's 's actions such that they exert greater control over the environmental outcomes and can solve tasks fewer steps. We lift the major limi- tation of prior work in this area scalability to nighh-imeessinal settings with contrastive successor representations. We formally prove that these representations estimate a similar notion of empowerment to that studied by prior work and provide a mechanism for optimizing Empirically, our proposed method outperforms prior methods synthetic benchmarks, and scales to Overcooked, cooperative game setting. Theoretically, our work connects ideas from information theory, neuroscience, and reinforcement learning, and charts path for representations play a critical role solving assistive problems."
Introduction
AI agents deployed in the real world should be helpful humans. When we know the utility function of the humans an agent could interact with, we can directly train assistive agents through reinforcement learning with the known human objective as the agent's s reward. In practice, agents rarely have direct access to scalar reward corresponding to human preferences (if such a consistent model even exists) [1], and must infer them from human behavior [2, 3]. This inference can be challenging, as humans may act suboptimally with respect to their stated goals, not know their goals, or have changing preferences [4]. Optimizing a misspecified reward function can have poor consequences [5].
alternative paradigm for assistance to train agents that are intrinsically motivated to assist humans, rather than directly optimizing a model of their preferences. An analogy can be drawn to a parent raising a child. good parent will empower the child make impactful decisions and flourish, rather than proscribing an "optimal" outcome for the child. Likewise, Al agents might seek to empower the human agents they interact with, maximizing their capacity change the environment [6]. In practice, concrete notions of empowerment can be difficult to optimize as an objective, requiring extensive modeling assumptions that don't 't scale well the high nigh-dimensional settings deep reinforcement learning agents are deployed in.
What is good intrinsic objective for assisting humans that doesn't require these assumptions? We propose notion of assistance based on maximizing the influence of the human's 's actions on the
Code: https://github. http:///wwwweeeeeee////eeeeeeeee hempweereeeeemmeeeeeeeeeeeee http://oeeeeetttccccerrrrrrreeee successor_eepreeentttiioo
Website: https: http::::://////wriing//// www.inngnghuunnsseee humans. gi thtb..b
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Without Empowerment
Empowerment
3
Figure I: We propose algorithm training assistive agents to empower human users the assistant should take actions that enable human users to visit a wide range of future states, and the human's actions should exert high degree of influence over the future outcomes. Our algorithm scales to high-dimensional settings, opening the door building assistive agents that need not directly reason about human intentions.
enviromment. This approach only requires one structural assumption: the AI agent is interacting with an environment where there notion of actions taken by the human agent a more general setting than the case where we model the human actions as the outcome of some optimization procedure, as inverse RL [7, 8] preference-based RL [9].
Prior work has studied various objectives for empowerment. For instance, Du et al. [6] approximate human empowerment the variance in the final states of random rollouts. Despite excellent results in certain settings, this approach can be challenging to scale to higher dimensional settings, and does not necessarily enable human users achieve the goals the want achieve. By contrast, approach exclusively empowers the human with respect to the distribution of (useful) behaviors induced by their current policy (which we term the effective empowerment), and can be implemented through a simple objective derived from contrastive successor features, which can then be optimized with scalable deep reinforcement learning (Fig. 1). We provide theoretical framework connecting our objective to prior work on empowerment and goal inference, and empirically show that agents trained with this objective can assist humans in the Overcooked environment [10] as well as more complex versions of the obstacle gridworld assistance benchmark proposed by Du et al. [6].
Our contribution is objective for training agents that intrinsically motivated to assist humans without requiring a model of the human's 's reward function. The approach, Empowerment via Successor Representations (ESR), maximizes the influence of the human's 's actions the environment, and, unlike past approaches for assistance without reward inference, is based on a scalable model-free objective that can be derived from learned successor features that encode which states the human may want to reach given their current action. By maximizing effective empowerment, our objective empowers the human reach the desired states, all states, without assuming human model. We analyze this objective in terms of empowerment and goal inference, drawing novel mathematical connections between time-series representations, decision-male making, and assistance. We empirically show that agents trained with our objective can assist humans in two benchmarks proposed by past work: the Overcooked environment [10] and an ibstacle-avoidance gridworld [6].
Related Work
Our approach broadly connects ideas from contrastive representation learning and intrinsic motivation to the problem of assisting humans.
Assistive Agents.
There are two lines of past work assistive agents that most relevant.

---

The first line of work focuses on the setting of an assistance game [2], where a robot (AI) agent tries to optimize a human reward of which it initially unaware. Practically, inverse reinforcement learning (IRL.) can be used in such a setting to infer the human's 's reward function and assist the human in achieving their goals [3]. The key challenge with this approach is that requires modeling the human's 's reward function. This can be difficult in practice, especially if the human's a behavior is not well-modeled by the reward architecture. Slightly misspecified reward functions can lead to catastrophic outcomes (i.e., directly harmful behavior in the assistance context) [11-13]. These concerns have prompted interest, under the umbrella term Al Aligmment, in ensuring agents are safe and adhere to human values [14-17]. Our approach avoids some of these issues by focusing on empowerment, a more general objective that does not require modeling the human's 's reward function.
The second line of work focuses on empowerment- like objectives for assistance and shared autonomy. Empowerment generally refers to a measure of an agent's ability to influence the environment [18, 19]. In the context of assistance, Du et al. [6] show one such approximation of empowerment (AvE) can be approximated in simple environments through random rollouts to assist humans. Meanwhile, empowerment-like objectives have been used in shared autonomy settings to assist humans with telcoperation [20] and general assistive interfaces [21]. A key limitation of these approaches for general assistance is they only model empowerment over one time step. Our approach enables more scalable notion of empowerment that can be computed over multiple time steps.
Intrinsic Motivation. Intrinsic motivation broadly refers to agents that accomplish behaviors in the absence of an externally-specified reward or task [22]. Common applications of intrinsic motivation in single-agent reinforcement learning include exploration and skill discovery [23-25], empowerment [18, 19], and surprise minimization [19. 26, 27]. When applied to settings with humans, these objectives may lead to antisocial behavior [5]. Our approach applies intrinsic motivation the setting of assisting humans, where the agent's s goal is an empowerment objective - to maximize the human's s ability to change the environment.
Information-theoreti Decision Making. Information-theerrtic approaches have seen broad appli- cability across unsupervised reinforcement learning [19, 23, 28]. These methods have been applied to goal-reaching [29]. skill discovery [24, 30-33], and exploration [25. 34, 35]. In the context of assisting humans, inforrition--teerrtiic methods have primarily been used to reason about the
human's 's goals or rewards [36-38].
Our approach is made possible by advances in contrastive representation learning for efficient estimation of the mutual information of sequence data [39]. While these methods have been widely used for representation learning [40, 41] and reinforcement learning [42-45], to the best of our knowledge prior work has not used these contrastive techniques for learning assistive agents.
3 The Information Geometry of Empowerment
We will first state general notion of an assistive setting, then show how an empowerment objective based on learned successor representations can be used to assist humans without making assumptions about the human following an underlying reward function. In Section 5, we provide empirical evidence supporting these claims.
3.1 Preliminaries
Formally, we adapt the notation of Hadfield- et al. [2], and assume a "robot" (R) and "human" (H) policy are training together in an MDP M = (S,An AR, R, P,7). The states 8 consist of the joint states of the robot and the human; we do not have separate observations for the human and robot. At any state s E S, the robot policy selects actions distributed according to (a
R. s) for a E Ar and the human selects actions from (H(a" s) for a Ah. The transition dynamics are defined by a distribution P(s' | ss, ss an) over the next state E given the current state s E S and actions a Ah and a E AR, as well as initial state distribution P(so). For notational convenience, we will additionally define random variables s to represent the state at time f, and a, ~ TR(' sx) and al TH( 5.) represent the human and robot actions time t, respectively.
Effective Empowerment. Our work builds on long line of prior methods that use information theoretic objectives for RI. Specifically, we adopt empoverment as an objective for training an assistive agent [6. 18, 46]. This section provides the mathematical foundations for empowerment,
3

---

as developed in prior work. We build on the prior work by (1) providing an information geometric interpretation of what empowernent does (Section 3.3) and (2) providing a scalable algorithm for estimating and optimizing notion of empowerment, the effective empowereepeeeee empowerment (Section 4.1).
The idea behind empowerment is to think about the changes that an agent can effect on world; an agent is more empowered if its actions effect a larger degree of change over future outcomes. Following prior work [18, 29, 46], we measure empowerment by looking at how much the actions taken now affect outcomes in the future. An agent with a high degree of empowerment exerts a high degree of control of the future states by simply changing its current actions. Like prior work, we measure this degree of control through the mutual information (s aa)) between the current action all and the future states s.. Note that these future states might occur many time steps into the future.
Empowerment depends on several factors: the environment dynamics, the choice of future actions, the current state, and other agents in the environment. Different problem settings involve maximizing empowerment using these different factors. In this work, we study the setting where a "human" agent and a "robot" agent collaborate in an environment, with the robot aiming to maximize the empowerment of the human. This problem setting was introduced in prior work [2, 6]. Compared with other mathematical frameworks for learning assistive agents [47], framing the problem in terms of empowerment means that the assistive agent need not infer the human's 's underlying intention, an inference problem that typically challenging [48, 49].
To define our empowerment objective, we introduce the random variable s*, corresponding to a state sampled K" Geom 1 - 7) steps into the future under the behavior policies "H and Tn- We will use p(s+ 8t) to denote the density of this random variable; this density is sometimes referred to as the discounted state occupancy measure. We will use mutual information measure how much the action ar at time & changes this distribution:
log P(S = St+k 5e = St, at oz)
(a :s [8)) A E Ber.erte.o
(1)
P(S 814k 5 8c)
Our overall objective is effective empowerment, { TR): the mutual information between the human's 's actions and the future states g+ while interacting with the robot:
DC 8(TH, TR. = Iaa
ss)
We call this objective effective empowerment rather than just empowerment to highlight that the mutual information in Eq. (2) is computed under the behavior policies मн and TR. This is in contrast to much of the prior work on empowerment, which focuses on the highest possible empowerment that could be achieved under some policy, rather than the actual realized empowerment [6, 18, 46). Focusing on the effective empowerment simplifies the problem of estimating empowerment since we only have access to samples from the current behavior policies, not arbitrary more powerful policies. In later sections, we will see this simplification is both empirically effective and theoretically justified.
Note that this objective resembles an RL objective: we do not just want to maximize this objective greedily at each time step, but rather want the assistive agents to take actions now that help the human agent reach states where it will have high empowerment in the future.
3.2 Intuition and Geometry of Empowerment
Intuitively, the assistive agent should aim to maximize the number of future that can be realized by the human's 's actions. We will mathematically quantify this in terms of the discounted state occupancy measure, n² (s+ s). An agent has a large empowerment if the future states for one action are very different from the future actions after taking a different action; i.c., when p(a, = a1;s* (s)) is quite different from p(a, s st) for actions d1 # a2. The mutual information (Eq. 1) quantifies this st).
degree of control: 1 [(a;:s"
One way of understanding this mutual information is through information geometry [50, 51, 51, 52]. For fixed current state St, assistant policy TR and human policy TH. each potential action o, that the human takes induces a different distribution over future states: p R (s St, az). We can think about the set of these possible distributions: [D (s+ 8,, a.) 0r E A). Figure (Lefi) visualizes this distribution on a probability simplex for 6 choices of action a, If we look at any possible distribution over actions, then this set of possible future distributions becomes a polytope (see orange polygon in Fig. 2 (Center)).

---

44%
((5)
0.4
00
600
0.4
p(22)
66
68
088
p(s))
28%
28%
(a) State marginal polytope
(b) Mutual information
(c) Maximizing empowerment
Figure 2: The Information Geometry of Empowerment, illustrating the analysis in Section 3.3. (Left) For a given state 8t and assistant policy TR. we plot the distribution over future states for choices of the human policy тH. In 3-state MDP, we can represent each policy vector lying the 2-dimensional probability simplex. We refer to the set of all possible state distributions as the state marginal polytope. (Center) Mutual information corresponds to the distance between the center of the polytope and the vertices that are maximally far away. (Right) Empowerment corresponds to maximizing the size of this polytope. For example, when assistive agent moves obstacle out of a human user's s way, the human user can spend more time at desired state.
Intuitively, the mutual information s(,;s |x)) in our empowerment objective corresponds to the size or volume of this state marginal polytope. This intuition can be formalized by using results from information geometry [53-55]. The human policy (ac 5t) places probability mass on the different points Figure 2 (Center). Maximizing the mutual information corresponds to "picking out" the state distributions that are maximally spread apart (see probabilities in Fig. 2 (Center)). To formalize this, define
p(s |x)) (x(a..s.) (p(st 8t. uz)]
as the average state distribution from taking the human's 's actions (see green square in Fig. 2 (Center)).
Remark 3.1. Mutual information corresponds to the distance between the average state distribution (Eq. 3) and the furthest achievable state distributions:
I(ay; ss) = -max DKu (planis st) p(s ss))
dmax.
(4)
This distance is visualized the black lines in Fig. 2. When talk about the "size" of the state marginal polytope, we are specifically referring to the length of these black lines (as measured with KL divergence).
This sort of mutual information is a way for measuring the degree of control that an agent exerts environment. This measure is well defined for any agent/policy; that agent need not be maximizing mutual information, and could instead be maximizing some arbitrary reward function. This point important in our setting: this means that the assistive agent estimate and maximize the empowerment of the human user without having to infer what reward function the human is trying maximize.
reward.funtion
Finally, we come back to our empowerment objective, which is a discounted sum of the mutual information terms that have been analyzing above. This empowerment objective says that the human is more empowered when this set has a larger size - i.c. the human can visit wider range of future state (distributions). The empowerment objective says that the assistive agent should act to try to maximize the size of this polytope. Importantly, this maximization problem is done sequentially: the assistive agent wants the size of this polytope to be large both at the current state and future states; the human's actions should exert a high degree of influence over the future outcomes both now and in the future. Thus, overall objective looks
of these mutual informations.
Not only does this analysis provides a geometric picture what empowerment is doing, also lays the work for formally relating empowerment to reward.
5

---

3.3 Relating Empowerment to Reward
In this section take aim at the question: when humans well-modeled optimizing a reward function, when does maximizing effective empowerment help humans maximize their rewards? Answering this question is important because for empowerment to be safe and effective assistive objective, it should enable the human to better achieve their goals. We show that under certain assumptions, empowerment yields provable lower bound on the average-case reward achieved by the human for suffiently long-horizon empowerment (i.e., 7 - 1).
For constructing the formal bound, we suppose the human is Boltzmann- noltzmann-rational [56, 57] with respect to some reward function R ~ R and rationality coefficient B. The distribution R could interpreted as prior over the human's 's objective, set of skills the human may try and carry out, population of humans with different objectives that the agent could be interacting with. Our quantity interest, the average-case reward achieved by the human with our empowerment objective, is given by
T (TH)
E R-R 80rppo
[VTH.TT (so)]
(5)
where Vin (5o) is the value function of the human policy TH under the reward function R when interacting with TR- Recalling Eq. (2), we will express the overall effective empowerment objective we are trying relate to Eq. (5) as
(TH. TR) = E r= "'Is"";a H. m.)
(6)
This notation is formalized in Appendix B.
The two key assumptions used in our analysis are Assumption 3.1, which states that the human will optimize for behaviors that uniformly cover the state space, and Assumption 3.2, which simply states that with infinite time, the human will be able to reach any state in the state space.
Assumption 3.1 (Skill Coverage). The rewards ~ are uniformly distributed over the scaled |8| -implex |S| such that:
(R+ DS ( ~ Unif(A 8I) = (1,ichlet(1111,..11
[S]
times
Assumption 3.2 (Ergodicity). For some TH, TR. have
p (s = so) > 0 for all E S,, (0,1).
(8)
Our main theoretical result is Theorem 3.1, which shows that under these assumptions, maximizing effective empowerment yields a lower bound on the (squared) average-case reward achieved by the human for sufficiently large In other words, for a sufficiently long empowerment horizon, the empowerment objective Eq. (2) is meaningful proxy for reward maximization.
Theorem 3.1. Under Assumption 3.1 and Assumption f.r.for sufficiently large 7 and any 3 0.
(,H, RR. 1/2 (B/e) (TH).
(9)
The proof is in Appendix B.1 To the best of our knowledge, this result provides the first formal link between empowerment maximization and reward maximization. This motivates us to develop scalable algorithm for empowerment maximization, which we introduce the following section.
Estimating and Maximizing Empowerment with Contrastive Representations
Directly computing Eq. (2) would require access to the human policy, which we don't 't have. There- fore, we want a tractable estimation that still performs well in large environments which are more difficult to model due to the exponentially increasing set of possible future states. To esttmree empowerment, we learn contrastive representations that encode information about which future states are likely to be reached from the current state. These contrastive representations learn to model mutual information between the current state, action, and future state, which we then use to compute the empowerment objective.

---

4.1 Estimating Effective Empowerment
To estimate this effective empowerment objective, we need way of learning the probability ratio inside the expectation. Prior methods such as Du et al. [6] and Salge et al. [18] rollout possible future states and compute measure of their variance proxy for empowerment, however this doesn't scale when the environment becomes complex. Other methods learn dynamics model, which also doesn't scale when dynamics become challenging to model [31]. Modeling these probabilities directly is challenging in settings with high-dimensional states, so we opt for an indirect approach. Specifically, we will learn representations that encode two probability ratios. Then, we will be able to compute the desired probability ratio by combining these other probability ratios.
Our method learns three representations:
Φ(s, a This representation can be understood as sort of latent-space model, predicting the future representation given the current state and the human's ' current action a as well as the robot's current action R
2. (s, ) -
This representation can be understood an uncontrolled model, predicting the representation a future state without reference to the current human action a This represen- tation is analogous value function.
( This a representation of a future state.
We will learn these three representations with two contrastive losses, one that aligns é(s,a", an) tt (i())) and one that aligns d'(s, a) (ss))
((s, a:) ((s max (( P() )| 6,0'
+ (((())))))((((())))))))))))
where the contrastive loss Le is the symmetrized infoNCE objective [39]:
N
e V:
v
L,((x;), (2o))
log
log
(10)
e M:
We have colored the index j for clarity. At convergence, these representations encode two probability ratios [28], which we will ultimately be able use to estimate empowerment (Eq. 2):
P(
= =s, = = H R = n
(ss a H (()((()) log
(11)
C1 P(5++ = 9)
P( = 814k St 8t, nt R
d' a ( log
(12)
C2 P(s = g)
Note that our definition of empowerment (Eq. is defined in terms of similar probability ratios. The constants C₁ and C2 will mean that our estimate of empowerment may be off by additive constant, but that constant will not affect the solution to the empowerment maximization problem.
4.2 Estimating Empowerment with the Learned Representations
To estimate empowerment, we will look at the difference between these two inner products:
0(54KK a (HHTTW( 0(s(* a (g)
P(s 1, H) log C₁ C2 Ci
log P(344K] log P(5 s) + log C2 + log P(3==R)
P(8++K 8, H) log
=log P(5+4K| (8)
Note that the expected value of the first term the conditional mutual information 1 [(5+ (Ss4-K;& a s). Our empowerment objective corresponds to averaging this mutual information across all the visited states. other words, our objective corresponds an RL problem, where empowerment corresponds to the expected discounted sum of these log ratios:
E(THATR) = ETH.TR
7 11(s H st)]
C2 Ci
R E ENgLR
(o(o R ,a M(or, B)) log
7

---

The approximation above comes from function approximation in learning the Bayes optimal repre- sentations. Again, note that the constants C₁ and C2 do change the optimization problem. Thus, to maximize empowerment we will apply RI. to the assistive agent (a 8) using reward function
r(s, a R)
(0( a aa alo,, () (g))
(13)
4.3 Algorithm Summary
Algorithm I: Empowerment via Successor Representations (ESR)
Input:
Human policy (H
Randomly initialize assistive agent policy (a s), and representations ((s, R. aa (H),, tss a²), and ()))). Initialize replay buffer 83 while not converged do
Collect trajectory of experience with human policy and assistive agent policy, store in replay buffer Update representations o(s, R. a (ss a"), and x(g) with contrastive losses Eq. (10). Update (a s) with RL using reward function r(s, R a (ø(s, a R an) d' (s, )) (T (b(g).
Return: Assistive policy (a 8).
We propose an actor-critic method for learning the assistive agent. Our method will alternate between updating these contrastive representations and using them estimate reward function (Fq. (13)) that is optimized via RI.. We summarize the algorithm in Algorithm 1. In practice, we use SAC [58] our RL algorithm. In our experiments, we will also study the setting where the human user updates their policy alongside the assistive agent.
obstacles
obstacles
30
20
Reward
20
10
10
50k
100k
150k
200k
250k
100k
200k
300k
400k
7 obstacles
10 obstacles
20
10
Reward
10
200k
400k
600k
500k
1M
1.5M
Step
Step
ESR (Ours)
AvE
Random
ESR-Norm
Goal Inference
Reward Inference
Figure 3: We apply our method to the benchmark proposed in prior work [6], visualized in Fig. 4a. The four subplots show variant tasks of increasing complexity (more blocks), (#11 SE). We compare against AvE [6], the Goal Inference baseline from [6] which assumes access to a world model, and Reward Inference [59] where recover the reward from learned q-value. These prior approaches fail on all except the easiest task, highlighting the importance of scalability.

---

Experiments
We seek to answer two questions with our experiments. First, does our approach enable assistance in standard cooperation benchmarks? Second, does our approach scale to harder benchmarks where prior methods fail?
Our experiments will use two benchmarks designed by prior work to study assistance: the obstacle gridworld [6] and Overcooked [10]. Our main baseline is AvE [6], a prior snpoowrrment-bseed method. Our conjecture is that both methods will perform well on the lower-dimensional grid- world task, and that our method will scale more gracefully the higher dimensional Overcooked environment. We will also compare against naive baseline where the assistive agent acts randomly.
(a) Obstacle Gridworld
(b) Cramped Room
(c) Coordination Ring
Figure 4: (a) The modified environment from Du al. [6] scaled N 7 blocks, and (b, c) the two layouts of the Overcooked environment [10].
5.1 Do contrastive successor representations effectively estimate empowerment?
We test our approach in the assistance benchmark suggested in Du et al. [6]. The human (orange) tasked with reaching a goal state (green) while avoiding the obstacles (purple). The AI assistant can move blocks one step at time in any direction [6]. While the original benchmark used N obstacles, we will additionally evaluate on harder versions of this task with N 5, 7, 10 obstacles. We show results in Fig. 3. On the easiest task, both our method and AvE achieve similar asymptotic reward, though our method learns more slowly than AvE. However, on the tasks with moderate and high degrees of complexity, our approach (ESR) achieves significantly higher rewards than AvE, which performs worse than a random controller. These experiments support our claim that contrastive successor representations provide an effective means for estimating empowerment, and hint that ESR might be well suited for solving higher dimensional tasks.
robot Āy (ESR)
human
Robot Adds Onion
Human Adds Onion
Empty Pot
Onions
Figure 5: In Coordination Ring, ESR agent learns to wait for the human to add an onion the pot, and then adds one itself. There another pot at the top which is nearly full, but the empowerment agent takes actions maximize the impact of the human's actions, and follows the lead of the human by filling the empty pot.

---

5.2 Does our approach scale to tasks with image-based observations?
Our second set of experiments look scaling ESR to the image-based Overcooked environment. Since contrastive learning is often applied to image domains, we conjectured that ESR would scale gracefully to this setting. We will evaluate our approach in assisting a human policy trained with behavioral cloning taken from Laidlaw and Dragan [60]. The human prepares dishes by picking up ingredients and cooking them on a stove, while the AI assistant moves ingredients and dishes around the kitchen. We focus on two environments within this setting: a cramped room where the human must pass ingredients and dishes through a narrow corridor, and a coordination ring where the human must pass ingredients and dishes around ring-shaped kitchen (Figs. 4b and 4c). As before, we compare with AvE as well as a naive random controller. We report results in Table On both tasks, we observe that our approach achieves higher rewards than AvE baseline, which performs no better than random controller. In Fig. 5, we show an example of one of the collaborative behaviors learned by ESR. Taken together with the results in the previous setting, these results highlight the scalability of ESR to higher dimensional problems.
Table I: Overcooked Results
Layout
ESR (Ours)
Reward Inference
AvE
Random
Asymmetric Advantages
72.00 + 5.37
60.33 + 0.26
36.71 + 1.71
59.36
Coordination Ring
8.40 + 0.69
5.96 + 0.20
5.69 + 0.93
6.02
Cramped Room
91.33 + 4.08
39.24 + 0.35
5.13 + 1.31
69.26
Discussion
One of the most important problems in AI today is equipping Al agents with the capacity to assist humans achieve their goals. While much of the prior work in this area requires inferring the human's 's intention, our work builds on prior work in studying how an assistive agent can empower a human user without inferring their intention. Relative to prior methods, we demonstrate how empowerment can be readily estimated using contrastive learning, paving the way for deploying these techniques on high-dimensional problems.
Limitations. One of the main limitations of our approach is the assumption that the assistive agent has access to the human's actions, which could be challenging to observe in practice. Automatically inferring the human's 's actions remains important problem for future work. A second limitation is that the method is currently an on-policy method, in the sense that the assistive agent has to learn by trial and error. A third limitation is that the ESR formulation assumes that both agents share the same state space. In many cases the empowerment objective will still lead to desirable behavior, however, care must be taken in cases where the agent can restrict the information its own observations, which could lead to reward hacking. Finally, our experiments do not test our method against real humans, whose policies may differ from the simulated policies. In the future, we plan to investigate techniques from off- off-policy evaluation and cooperative game theory to enable faster learning of assistive agents with fewer trials. We also plan to test the ESR objective in environments with partial observability over the human's 's state.
Safety risks. Perhaps the main risk involved with maximizing empowerment that it may be at odds with human's 's agents goal, especially in contexts where the pursuit of that goal limits the human's 's capacity to pursue other goals. For example, a family choosing to have a kid has many fewer options over where they can travel for vacation, yet do not want assistive agents to stymie families from having children.
One key consideration is whom should be empowered. The present paper assumes there is a single human agent. Equivalently, this can be seen as maximizing the empowerment of all exogenous agents. However, it is easy to adapt the proposed method to maximize the empowerment of a single target individual. Given historical inequities the distribution of power, practitioners must take care when considering whose empowerment to maximize. Similarly, while we focused on maximizing empowerment, trivial to change the sign so that an "assistive" agent minimizes empowerment. One could imagine using such a tool in policies to handicap one's political opponents.
10

---

Acknowledgments. We would like to thank Micah Carroll and Cameron Allen for their helpful feedback, well as Niklas Lauffer for suggesting JaxMARL. We especially thank the fantastic NeurlPS reviewers for their constructive comments and suggestions. This research was partly supported by ARI. DCIST CRA W911NF-17-2-018 and ONR N00014-22-1-2775 as well as NSF HCC 2310757, the Jump Cocosys Center, Princeton Research Computing, and the Dol through the NDSEG Fellowship Program.
References
[1] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open Problems and Fundamental Limitations of Reinforcement Learning From Human Feedback. arXiv:2307.15211, 2023.
[2] Dylan Hadfield-Menell, Stuart J. Russell, Pieter Abbeel, and Anca Dragan. Cooperative Inverse Reinforcement Learning. Neural Information Processing Systems, 29, 2016.
[3] Dylan Hadfield- Smitha Milli, Pieter Abbeel, Stuart J. Russell, and Anca Dragan. Inverse Reward Design. Neural Information Processing Systems, 30, 2017.
[4] Micah Carroll, Dylan Hadficld-Menell, Stuart Russell, and Anca Dragan. Estimating and Penalizing Preference Shift in Recommender Systems. ACM Conference on Recommender Systems, RecSys '21, pp. 661-667, 2021.
[5] Alexander Matt Turner, Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli. Optimal Policies Tend to Seek Power. Neural Information Processing Systems, 2023.
[6] Yuging Du Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anca Dragan. AvE: Assistance via Empowerment, Neural Information Processing Systems, volume 33, pp. 4560-4571, 2020.
[7] Stuart Russell. Learning Agents for Uncertain Environments. Annual Conference on Computa-
lional Learning Theory, pp. 101-103, 1998.
[8] Saurabh Arora and Prashant Doshi. A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress. Artificial Intelligence, 297:103500, 2021.
[9] Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A Survey of Preference-Based Reinforcement Learning Methods. Journal of Machine Learning Research, 18(136):1-46, 2017.
[10] Micah Carroll, Rohin Shah, Mark K. Ho, Thomas Griffiths, Sanjit A. Seshia, Pieter Abbeel, and Anca Dragan. On the Utility of Learning About Humans for Human-Al Coordination. Neural Information Processing Systems, 2019.
[11] Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The Effects of Reward Misspecifica- tion: Mapping and Mitigating Misaligned Models. International Conference on Learning Representations,
2022.
[12] Jeremy Tien, Jerry Zhi- Yang He, Zackory Erickson, Anca Dragan, and Daniel S. Brown. Causal Confusion and Reward Misidentification in Preference-Bast Reward Learning, Inter- national Conference on Learning Representations, 2023.
[13] Cassidy Laidlaw, Shivam Singhal, and Anca Dragan. Preventing Reward Hacking With Occu-
pancy Measure Regularization, arXiv:2403.00 0315 2024.
[14] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in AI Safety. arXiv: 1.Xiv:1606.000 666606665. 2016.
[15] Richard Ngo, Lawrence Chan, and Sören Mindermann. The Alignment Problem From a Deep
Learning Perspective. International Conference on Learning Representationss 2024.
[16] Jessica Taylor, Eliezer Yudkowsky, Patrick Va and Andrew Critch. Alignment for Advanced Machine Learning Systems. S. Matthew Liao, editor, Ethics of Artificial Intelligence,
pp. 342-382. Oxford University PressNew York, 1 edition, 2020.
[17] Simon Zhuang and Dylan Hadfield-Menell Consequences Misaligned AI. Neural Information Processing Systems, volume 33, pp. 15763-15773, 2020.
[18] Christoph Salge, Cornelius Glackin, and Daniel Polani. Empowerment-an Introduction. Guided Self-Orraaization: Inception, 110x1::13100.183)67-11111 2014.
11

---

[19] IIdefons Magrans de Abril and Ryota Kanai. A Unified Strategy for Implementing Curiosity and Empowerment Driven Reinforcement Learning, arXiv:1806.06505 2018.
[20] Sean Chen, Jensen Gao, Siddharth Reddy, Glen Berseth, Anca D. Dragan, and Sergey Levine.
ASHA: Assistive Teleoperation via Nluma--i-tth--one Reinforcement Learning. International Conference on Robotics and Automation, 2022.
[21]
Siddharth Reddy, Sergey Levine, and Anca Dragan. First Contact: Unsupervised Human-
Machine Co-Adaptation via Mutual Information Maximization. Neural Information Processing
Systems, 35:31542-31556, 2022.
[22] Andrew G. Barto. Intrinsic Motivation and Reinforcement Learning Gianluca Baldassarre and Marco Mirolli, editors, Intrinsically Motivated Learning in Natural and Artificial Systems, pp. 17-47. Springer, 2013.
[23]
Arthur Aubret, Laetitia Matignon, and Salima Hassas. A Survey on Intrinsic Motivation in Reinforcement Learning. Entropy, 2023.
Need: Learning Skills Without Reward Function. International Conference on Learning [24]
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity Is All You
Representations, 2019.
[25] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by Random
Network Distillation. International Conference on Machine Learning, 2023.
[26] Karl Friston. The Free-Eneee Eacce--rrgyyy Principle: A Unified Brain Theory? Nature Reviews Neuro- science, 11(2):127-138, 2010.
[27] Glen Berseth, Daniel Geng, Coline Devin, Nicholas Rhinehart, Chelsea Finn, Dinesh Jayara-
man, and Sergey Levine. SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments. International Conference on Learning Reprexentations, 2019.
[28] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On Variational Bounds of Mutual Information. International Conference on Machine Learning, 2019.
[29] Jongwook Choi, Archit Sharma, Honglak Lee, Sergey Levine, and Shixiang Shane Gu. Varia- tional Empowerment as Representation Learning for Coal-Conditioned Reinforcement Learning. International Conference on Machine Learning, pp. 1953-1963, 2021.
[30] Shakir Mohamed and Danilo Jimenez Rezende. Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning. Neural Information Processing Systems, volume 28, 2015.
[31] Tobias Jung, Daniel Polani, and Peter Stone. Empowerment for Continuous Agent Environment Systems. Adaptive Behavior, 19(1):16-39, 2011.
[32]
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery. arXiv:2202.0016 2022.
[33] Seohong Park, Jongwook Choi, Jaekyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz- Constrained Unsupervised Skill Discovery. International Conference on Learning Representa- tions, 2021.
[34] Susanne Still and Doina Precup. An Information- Theorrtic Approach to Buriosity-Driver Reinforcement Learning. Theory in Biosciences, 131:139-144; 2012.
[35] Nikolay Nikolov, Johannes Kirschner, Felix Berkenkamp, and Andreas Krause. Information- Directed Exploration for Deep Reinforcement Learning. International Conference on Learning Representations, 2019.
[36] Erdem Biyik, Dylan P. Losey, Malayandi Palan, Nicholas C. Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning Reward Functions From Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences. Int. J. Robotics Res, 2022.
[37] Vivek Myers, Erdem Biyik, Nima Anari, and Dorsa Sadigh. Learning Multimodal Rewards From Rankings. Conference Robot Learning, pp. 342-352 442-352,40 2021.
[38] Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian Active Learning for Classification and Preference Learning. arXiv:11 2011.
[39] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning With Contrastive Predictive Coding. arXiv: 1807 1807183744 003748, 2018
[40] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A Simple Frame- work for Contrastive Learning of Visual Representations. International Conference on Machine Learning, volume 1 19 of Proceedings of Machine Learning Research, pp. 1597-1607, 2020.
12

---

[41]
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised Feature Learning via Non-Parametric Instance Discrimination. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3733-3742, 2018.
[42] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive Unsupervised Repre- sentations for Reinforcement Learning. International Conference on Machine Learning, pp. 5639-5650, 2020.
[43] Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R. Salakhutdinov. Contrastive Learning as Goal-Conditiones Reinforcement Learning. Neural Information Processing Systems, volume 35, pp. 35603-35620, 2022.
[44] Peter Dayan. Improving Generalisation for Temporal Difference Learning: The Successor Representation. Neural Computation, (14):613-6244 1993.
[45] I. Momennejad, E. M. Russek, J. H. Cheong. M. M. Botvinick, N. D. Daw, and S. J. Gershman. The Successor Representation in Human Reinforcement Learning. Nature Human Behaviour,
1(9):680-692, 69000-6992200 2017.
[46] Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv. Empowerment: A Universal Agent-Centir Measure of Control. IEEE Congress on Evolutionary Computation, volume 1, pp. 128-135, 2005.
[47] Siddharth Reddy, Anca D Dragan, and Sergey Levine. Shared Autonomy via Deep Reinforce-
ment Learning. arXiv: 1.Xiv:1802.011 802.01744 2018.
[48] Nathan D. Ratliff. J. Andrew Bagnell, and Martin A. Zinkevich. Maximum Margin Planning. International Conference on Machine Learning ICML '06. pp. 729-736, 2006.
[49] Pieter Abbeel and Andrew Y. Ng. Apprenticeship Learning via Inverse Reinforcement Learning.
International Conference Machine Learning, p. 1, 2004.
[50] Thomas M Cover. Elements of Information Theory. John Wiley Sons, 1999.
[51] Nihat Ay, Jürgen Jost, Hông Vân Lê, and Lorenz Schwachhöfer. Information Geometry, volume 64 of Ergebmisse der Mathematik und ihrer Grenzgebiete 34. Springer International Publishing, 2017.
[52] Frank Nielsen. An Elementary Introduction to Information Geometry. Entropy, 22(10):1100, 2020.
[53] Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. The Information Geometry of
Unsupervised Reinforcement Learning. International Conference on Learning Representations, 2022.
[54] Robert G. Gallager. Source Coding With Side Information and Universal Coding. 1979.
[55] Boris Yakovlevich Ryabko. Coding of Source With Unknown but Ordered Probabilities. Problems of Information Transmission, 1542)1134-138, 1979.
[56) R. Duncan Luce. Individual Choice Behavior. Individual choice behavior. John Wiley, 1959.
[57] Brian D Ziebart, Andrew Maas, Andrew Bagnell, and Anind K Dey. Maximum Entropy Inverse Reinforcement Learning. AAAI Conference on Artificial Intelligence, 2008.
[58] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft Actor- Critic Algorithms and Applications. arXiv: 12.05905, 2019.
[59] Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. IQ- Learn: Inverse Soft-Q Learning for Imitation. Neural Information Processing Systems, vol- ume 34. pp. 4028-4039) 2021.
[60] Cassidy Laidlaw and Anca Dragan. The Boltzmann Policy Distribution: Accounting for System- atic Suboptimality in Human Models. International Conference Learning Representations, 2022.
[61] Diederik P. Kingma and Jimmy Ba. 411iv:1412.6980, 2017.
Adam: A Method for Stochastic Optimization.
[62] Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Gardar Ingvarsson, Timon Willi, Akbir Khan, et al. JaxMARL: Multi-Agent RL Environments in JAX. International Conference on Autonomous Agents and Multiagent Systems, 2024.
[63] Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. Rectifier Nonlinearities Improve Neural Network Acoustic Models. International Conference on Machine Learning, volume 30, p. 3, 2013.
13

---

[64] Tongzhou Wang and Phillip Isola. Understanding Contrastive Representation Learning Through Alignment and Uniformity on the Hypersphere. International Conference on Machine Learning. 2020.
[65] Benjamin Eysenbach, Vivek Myers, Ruslan Salakhutdinov, and Sergey Levine. Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inference. Neural Information Processing Systems, 2024.
14

---

A Experimental Details
We ran all our experiments on NVIDIA RTX A6000 GPUs with 48GB of memory within an internal cluster. Each evaluation seed took around 5-10 hours complete. Our losses (Eqs. 10 and 13) were computed and optimized in JAX with Adam [61]. We used a hardware--aceelerated version of the Overcooked environment from the JaxMARL package [62]. The experimental results described Section 5 were obtained by averaging over 5 seeds for the Overcooked coordination ring layout, 15 for the cramped room layout, and 20 for the obstacle gridworld environment. Specific hyperparameter values can be found in our code, which is available at https://g /ttt:///ghhtt thub. com/ vivekmyers/ empowerment.....mmmmme v http:///eeeeeeeettoooooooooooooo reacessor.eppresaaatiioom
A.1 Network Architecture
In the obstacle grid environment, we used network with convolutional and 2 fully connected layers and SiLU activations. In Overcooked, we adapted the policy architecture from past work [4], using 3 convolutional layers followed by MLP layers with Leaky ReLU activations [63]. We concatenate in R and a to the state one-hot encoded channels, i.e. if the action is 5, 6 additional channels will be concatenated to the state with all set to except the channel which is set Is.
B Theoretical Analysis of Empowerment
To connect our effective empowerment objective to reward, we will extend the notation Section 3.1 to include a distribution over possible tasks the human might be trying to solve, R., such that each R R defines distinct reward function R : S - R. We assume aR tries to maximize the d-discouteed effective empowerment of the human, defined as
( TR) E 1
('((sq;a 5s)
(Eq. 6)
for
2 [sc for k Geom - }
(14)
We additionally define to be the full history of states up time t and a." to be the full history of human actions up to time t,
5. {s, 1
a; = {a 1
(15)
Then, 9, is the full history of states and past human actions up to time 4,
= 5, H 5;
(16)
Note that the definition of empowerment in Eq. (6) differs slightly from the original construction Eq. (2) 12)-wa we condition on the full history of human actions, not just the most recent one. This distinction becomes irrelevant practice if our MDP maintains history the state, in which case we can equivalently use in place of št.
Mcanwhile, for any fixed mn and 3 > 0, the human is Boltzmann-ratione with respect to the robot's policy:
Th (a
5e) x exp(3QTT
TH (st, R27
aH)
(17)
where Q (st, H) E Ex=0 ntth R(544.))
St, az
H
(18)
Equivalently, we can define the human's 's (soft) Q-function and value as
R (St, H M(ss)) + 7B TH (st+1) 8z, az
for V.H. (st) E R(ss) + + R(s(+1) + R($(42) +... st, 1t
(19)
The overall human objective is to maximize the expected soft value:
Je (mH) = E R~R [VTH.TRR (su)]
(Eq. 5)
50~p0
15

---

Note that this definition of TH depends on R and RR and is bounded 0 S J (TH) S 1. As in the CIRL setting [2]. we assume robot is unable access the true human reward R R. One way to think of the robot's s task is as finding a Nash equilibrium between the objectives Eq. (6) and the human best response in Eq. (17).
For convenience, we will also define a multistep version of "RT
R,y (st, oz E+ k
Σ K=
R((++)) 8t: oz
(20)
B.1 Connecting Empowerment to Reward
Our approach will be to first relate the empowerment (influence of oz s to the mutual information between a and the reward R.
Then, will connect this quantity notion of "advantage" for the human (Eq. 27), which in turn can be related to the expected reward under the human's 's policy. In its simplest form, this argument will require assumption over the reward distribution:
Assumption 3.1 (Skill Coverage). The rewards R ~ R are uniformly distributed over the scaled |S| simplex |S| such that:
(R+
1 SS (+ ~ Unif(A S1) Dirichlet (I........)
[S] rimes
In other words, Assumption 3.1 says our prior over the human's reward function is uniform with zero mean. This not the only prior for which this argument works, but for general R we will need correction term to incentivize states that are more likely across the distribution of R. Another way to view Assumption 3.1 is that the human is trying to execute diverse "skills" 2 ~ Unif(
We also assume ergodicity (Assumption 3.2). In the special case of an MDP that resets to some distribution with full support over S, this assumption automatically satisfied. Assumption 3.2 (Ergodicity). For some TH, TR. have
pmu.an (s" s so) 0 for all E S,7 (0,
(8)
Our main result connects empowerment directly to (lower bound on) the human's expected reward. Theorem 3.1. Under Assumption 3.1 and Assumption 3.2, for sufficiently large 7 and any 3 > 0,
En TR. (3/e) J (TH).
Theorem 3.1 says that for long enough horizon (i.e., y close to 1), the robot's empowerment objective will lower bound the (squared, MaxEnt) human objective.
We make use of the following lemmas in the proof.
Lemma B.I. For Geom - 7) and any K 2 0.
H . lim inf [ss]; dt ..... a
(c) < T(R;
a (,))-
H
(21)
Proof. For sufficiently large 7, st will approach the stationary distribution of P for a fixed TH. TR. irrespective of 5, and aH H
a from Assumption 3.2. So,
lim inf [(s);; a. a
(x) < 7 lim s7 H H K+
5r
(22)
Since each R, RR. 7 defines human policy कн via Eq. (17), we can express the dependencies as the following Markov chain:
à
R
lim s2.
(23)
Applying the data processing inequality [50], we get
Ii
7( lim 8e
a ....; a
H s
(R:
at
4 5.),
.H
(24)
from which Eq. (21) follows.
16

---

Lemma B.2. Suppose we have k logits, denoted by the map a {1...k) + [0, 11. For any 3 > 0, we can construct the (softmax) distribution
Pa(i) x exp()a(i))
Then,
1 2 c
H(ps) 2 log h:
(25)
Proof. We lower bound the "worst-case" of the RHS, a (1.0....,0) 0):
(1 n) log H(ps)
e log k+e
e
k+t
h: + e
k e
(k + e 1)log (h e
1) e log(e
+ c -
c (og(
log(k E² -))
k e -
2 log k -B8/))
(26)
Lemma B.3. For any t and 2 0.
H T(R: of
H ar+
me) S lim 7-1
3 (st, of
B
d x)))
(27)
Proof. Denote by a
.......u.. âr+k= ~ Unif(,A a sequence of K random actions. From Lemma B.2:
I(R:a H ..... att (.) H(a H
H
H
(.) H(a,"
H a R,,c)
..... at* k
log K Al) (H(a," H
R.,
a+*
lim E
Q
(st, df H
a+ K
H
. (st, of ..... a x x))
H
(28)
where the last inequality follows from Lemma B.2 and Q (...) < 1. We also have
SSQR S TH. (St, aH .......... a k) I
R
QH. (Sp, H a.tk) S 1,
R,7
(29)
which lets us conclude from Eq. (28) that
I(R: H
H *
3 [ (,) <
(st, at
H )))
(Eq. 27)
We can now prove Thcorem 3.1 directly by combining Lemmas B. and B.3.
Proof of Theorem 3.1. Simplifying the limit in Eq, (9), we get
lim inf Cy TR) I lim 7-1
"T(s"
de H
(z)
7-1
lim inf [(5];
a
(5)
(chain rule)
7-11
< T(R; az
H
.....
atk
H
3t)
(Lemma B.
x))) 2
lim
TR
(8t, H
(Lemma B.3)
n1,7
7-1
e
(95%% (TH)
2
lim 7-1
(30)
follows that for sufficiently large 7
ET RR) 1/2 < (3/c) Jf (TH).
(Eq. 9)
O
17

---

2 obstacles
5 obstacles
30
20
Reward
10
10
50k
100k
150k
200k
250k
100k
200k
300k
400k
7 obstacles
10 obstacles
20
10
Reward
15
10
5
200k
400k
600k
500k
1M
1.5M
Step
Step
ESR (Ours)
ESR (-a
Figure 6: We evaluate our method with and without conditioning on the robot action R Conditioning aids learning significantly, which theorize because it removes uncertainty in the classification.
C Additional Ablations and Qualitative Results
In this section we evaluate additional ablations and qualitative results for the ESR method.
C.1 Learning Representations without the Robot Action
In our estimation of empowerment (Eq. 12) we supply the robot action a when learning both & and 6, however, the theoretical empowerment formulation in Section 3.3 does not require it.
To evaluate the impact of including aR we run an additional ablation without on the gridworld environment, shown in Fig. 6. This ablation shows that the inclusion of a is most impactful in more challenging (higher number of boxes) environments. We hypothesize that conditioning the representations the robot action reduces the noise in the mutual information estimation, and also reduces the difficulty of classifying true future states.
D Greedy Empowerment Policy
All of experiments have used Soft-Q learning to leam policy from the empowerment estimation. Here, we additionally study a greedy empowerment policy which takes the most empowering action each step. We model this by setting the q-learning gamma 0 to fully discount future rewards.
Results for this ablation are shown in Fig. 7. Unsurprisingly, the greedy optimization vastly underper- forms the policy with y 0.9.
18

---

2 obstacles
5 obstacles
30
20
Reward
20
10
10
50k
100k
150k 200k
250k
200k
400k
600k
7 obstacles
10 obstacles
20
10
Reward
15
10
5
200k
400k
600k
500k
1M
1.5M
Step
Step
ESR (Ours)
ESR (Greedy)
Figure 7: We compare a greedy policy (7 = 0) against our standard policy (y = 0.9).
D.1 ESR Training Example
In Fig. 8, we show the mutual information during training of the ESR agent in the gridworld environment with 5 obstacles. he mutual information quickly becomes positive and remains so throughout training. As long as the mutual information is positive, the classifier is able to reward the agent for taking actions that empower the human.
E
Simplifying the Objective
The reward function in Eq. (13) is itself random variable because depends future states g. This subsection describes how this randomness can be removed. To do this, we follow prior work [64, 65] in arguing that the learned representations (g(g) follow Gaussian distribution:
Assumption E.1 (Based on Wang and Isola [64]). The representations of future states (g(g) learned by contrastive learning have a marginal distribution that Gaussian:
P(t) = / P(g)8(( = +())) dg 4 1).
(31)
With this assumption, we can remove the random sampling of from the reward function. We start by noting that the learned representations tell us the relative likelihood of seeing future state Eq. (12)). Assumption E. will allow us to convert these relative likelihoods into likelihoods.
Ep an [r(s R) Ep(st)
P(s ".at"s r(s, R P(st)
=Epstt) CGellaa My "") d(o))
"rsssss (o)o(otrrsss)t)))) at)
= =CEEp-P(olot) [oolaaR Hy (0(8, a ( R, (()
19

---

Free State
Msn a "ac,aa.a*)) p(8,
9.10
0.05
6.0% 16 Hb.0% 21.0% 2..0%. 0.400.0
0.02
0
Trapped State
(ss, a a²) (s5 a
9.10
0.0
ho 1 22 b.. 0.00
02
Mutual Info
Avg. # Reachable States
0.25-
5.0
Aog. Reachable States
4.5-
0.15
4.0
0.10
3.0
2.5-
0.00
2.0
20
40
60
8a
100
100
Epoch
Epoch
Reward
10-
8-
Reward
3-
60
Epoch
Figure Visualizing training empowerment in 5x5 Gridworld with obstacles. Our empowerment objective maximizes the influence of the human's actions on the future state, preferring the state where the human can reach the goal to the trapped state. This corresponds to maximizing the volume of the state marginal polytope, which is proportional to the number of states that the human can reach from their current position. To visualize the representations, we set the latent dimension to instead of 100.
20

---

Ci R aa o() 7
(2xjurse 11l11i+t0 "a de
-C R aat) (),,at)) "dlldoaaa "a"""
/ (26ju7ze +1 "n 3llocs," " 12 dib
= =Ci R adt))) o(s,a
=Gezllo(aa 1 (o(s, R ah)) - o(s,oR)) T pls a a (32)
This simplification may be attractive in in where the computed empowerment bonuses have high variance, or when the empowerment horizon is large (i.e., y + 1, as in Section 3.3). Empirically, we found this version of the objective to be less effective in practice due to the additional representation structure required by Assumption E.1.
21

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Yes, our goals in the abstract and introduction are well-specified and accurately reflect the rest of the paper and the experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Yes, we have a limitation subsection that addresses the core limitations of the approach.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer: [Yes]
Justification: Yes, we clearly cite our assumptions and prior work that we are building on for theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, fully describe how reproduce our method by detailing the different contrastive networks, the exact loss functions we use, the environments we train on, and the human policies we are training against.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
23

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
Yes, our code is released at https: /ao//moon 4open. science/rrnceee
esr- 7e94
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nig) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. httppuulccggguueeee httii//ggiides/odeuubbiiiii aessonooiicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, all of the training details are included depth to help understand the results, and also the code containing all of this is released.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: Yes standard error bars are included and explained, and the number of seeds are given.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we provide the experimental details including the compute resources needed reproduce the experiments the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //te:://ene...eeeeeee ct/ccc//p//o wc//puulic//etiieeee htw.:e//tthiisiiieeeeeeeeeeeee
Answer: [Yes]
Justification: Yes, comply to the Code of Ethics in every way.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We extensively discuss various safety risks, and how empowerment plays into that, in the Discussion section of the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The data and models are not at risk of misuse, because they are in gridworld and Overcooked, and policies trained on these environments do not pose a safety issue.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Yes, we cite all the assets used, including code for environments, experiments, and algorithms, and graphics describing the environments. We additionally credit the authors of the behavioral cloning policies we use as human models, link to their repository/license, and mention that they use MIT License.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
26

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code that we release is well documented and the license is provided alongside the code repository.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper involves neither crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "7vLQ2X0XDN", "reviewer": "Reviewer_FYis", "summary": "This paper studies the problem of human-AI collaboration. The goal is to train an cooperative policy that can work with human together in the environment to achieve a shared goal.\n\nThe key idea of this paper is to maximize the influence of the human's actions on the environment, which is called empowerment. The paper provides an information geometric interpretation of empowerment and develops a algorithm based on SAC for estimating and optimizing empowerment. This method does not require to infer human's reward model.", "questions": "* What's the motivation by sampling k from a Geometric distribution in Line 136?\n* IIUC, the learned policy will \"overfit\" to the human policy which it is trained with? How to scale up the system to utilize the (offline) data from different human policy with different skills / characteristics / optimality?\n* Is there any measurement available to measure the \"human-compatibility\" during training? That is, will the human subject feel better with some robot policies against other robot policies even though the learning converge in the same speed?\n* Some clarity issues.\n  * Typo Line 268 \"limit’s\".\n  * No closing parenthesis in Line 218.\n  * What is \"this set\" in Line 142 (if the reader don't read caption of Fig2)", "limitations": "The limitations are well addressed.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1. Code is provided.\n2. The method removes the need to infer human reward function, making it scalable to high-dim space.\n3. The method is theoretically grounded and is simple to implement.", "weaknesses": "* No experiments involving real human subjects are conducted. \n* As suggested by the limitation, it's infeasible to train such a policy with a real human subject, especially in the task like overcooked which takes 1.5M steps to train...\n* The paper only compares to one baseline AvE. As the overcooked is a widely used environment studying human-robot collaboration, more results are welcome.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe thank the reviewer for the comments and suggestions for improving the paper. It seems like the reviewer's main suggestions were to add additional baselines, which we have done by adding comparisons with a goal inference method and a reward inference method. **Together with the discussion below, does this fully address the reviewer's concerns?**\n\nKind regards,\n\nThe authors.\n\n> The paper only compares to one baseline AvE. As Overcooked is a widely used environment studying human-robot collaboration, more results are welcome.\n\nWe have added a goal inference baseline (Du et al., 2021) to the obstacle environment and a reward inference baseline using IQ-Learn (Garg et al., 2021) for both the obstacle environments and the overcooked settings. Our method outperforms both of these baselines across the environments studied. We also add additional ablations of our approach (contrastive loss and parameterization). The two ablations are with the norm and diff variants of our reward. The norm reward computes the reward as the difference of norms between phi(s,a) and phi(s). The diff reward computes the reward as the norm of the difference between phi(s, a) and phi(s). These experiments are included in the attached PDF. Note that a key difference between our evaluations and prior work that studies assistance in Overcooked (e.g., Carroll et al. (2019)) is that we do not assume any access to the human reward *or* a model of the environment / possible reward structures.\n\n\n> Is there any measurement available to measure the \"human-compatibility\" during training?  \n\nThanks for this suggestion! This paper is primarily mathematical and algorithmic and nature, and so we'll provide an mathematical/algorithmic answer: Eq. 6 says that we can measure the degree of (minimax) compatibility via the mutual information. We have run an additional experiment to plot this mutual information throughout training (see Fig. 4 in rebuttal PDF). Visualizing the learned agent, we see that the agent does indeed become more helpful as this mutual information increases.\n\nOf course, the gold standard in human-AI interaction is human user studies, which are beyond the scope of this mathematical/algorithmic paper.\n\n> IIUC, the learned policy will \"overfit\" to the human policy which it is trained with? How to scale up the system to utilize the (offline) data from different human policy with different skills / characteristics / optimality?\n\nAs with any machine learning algorithm, there is a risk of overfitting when learning from limited data, and learning from a larger quantity of data may mitigate this risk. While our study focuses on the online learning of collaborative policies, the contrastive empowerment objective could also be applied to an offline dataset. In the case where diverse offline data is limited, it may be beneficial to initialize this contrastive policy with a large pretrained model that can impose strong inductive biases on the successor representations. Future work could also explore how multi-modal models (e.g., Chen et al., 2020) combined with our method to further boost performance\n\n> What's the motivation by sampling k from a Geometric distribution in Line 136?\n\nThe choice of distribution for $k$ dictates the ``horizon'' of the human's empowerment: do we want the human's actions to have a high influence over the outcomes in the next hour, or over the outcomes in the next day. For simplicity, we used a Geometric distribution to be consistent with RL standards (where we care about rewards accumulated under a geometric distribution) and set the parameter to the same value as used by the underlying RL algorithm.\n\n> No experiments involving real human subjects are conducted. \n\nWe have revised the paper to mention this limitation in the conclusion. Many prior papers also aim to establish the algorithmic foundations for human-AI algorithms before proceeding with human studies (Chan et al., 2019; He et al., 2023; Ngo et al., 2024; Pan et al., 2022; Ratner et al., 2018; Robertson et al., 2023; Zhuang & Hadfield-Menell, 2020).\n\n> it's infeasible to train such a policy with a real human subject, especially in the task like overcooked which takes 1.5M steps to train…\n\nYes, our method needs a large quantity of human data. However, we could apply our method in the offline setting where we have large scale existing datasets for this purpose (see e.g., Xie et al. (2018)). We could then fit the ESR features to the dataset and use the empowerment reward with an offline RL algorithm.\n\n> Typo Line 268 \"limit’s\".\n\nWe have fixed this.\n\n> No closing parenthesis in Line 218.\n\nWe have fixed this.\n\n> What is \"this set\" in Line 142 (if the reader don't read caption of Fig, 2)\n\nWe have clarified this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E049qEWQRw", "reviewer": "Reviewer_PN73", "summary": "The paper addresses the problem setting of human-agent collaboration where the agent learns to empower human decision-making to exert greater control over the environment. By connecting empowerment to reward maximization, the paper proposes the ESR method, which learns an intrinsic reward function based on learned representations of future states. Experiments demonstrate that ESR outperforms the AvE baseline in a gridworld task and Overcooked.", "questions": "1. L160 refers to \"the CIRL setting\", but CIRL is never defined. What does CIRL stand for?\n\n1. As stated in the limitations, ESR assumes access to the human actions to learn the empowerment reward. Can ESR work with noisy predictions of human actions? \n\n1. What is the performance of methods that assume access to the human intent via the task reward function in Overcooked? Are the results in Figure 5 directly comparable to those in Figure 4 from the Overcooked paper? \n\n1. Is the assumption that the agents share the same state space central to the empowerment learning objective? Is it possible to implement ESR if both agents operate from separate egocentric views?", "limitations": "Yes, the paper discusses the limitations and safety risks in Section 6.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. To the best of my knowledge, the ESR method is novel. ESR provides the new insight of connecting empowerment and reward maximization. This allows using empowerment as an intrinsic reward for training agents with RL.\n\n1. The paper shows ESR greatly outperforms the AvE baseline on more complex obstacle gridworlds and in Overcooked. In more complex environments, AvE is only able to achieve near-random performance. Results are also validated over multiple random seeds. \n\n1. The code is provided to reproduce results.", "weaknesses": "1. The paper only shows results on 2 of the Overcooked environments. This decision is not justified and with 15-20 seeds on the main Overcooked results, it is likely not an issue of sufficient compute to run these experiments. Even if the performance is worse on the remaining environments, the paper should also report numbers on the full Overcooked setting with \"Asymmetric Advantages\", \"Forced Coordination\" and \"Counter Circuit\".\n\n1. While the method addresses the problem of coordination without knowledge of the human's goal, the experiments demonstrating this are contrived. In Overcooked, the goal is fixed per task setting and is serving a dish. While ESR outperforms AvE, it greatly underperforms baselines from population-based training in [1]. This shows the large gap in empowerment versus the underlying objective of the task. The paper does not show results in a setting where the goals and preferences of the human are complex to communicate.\n\n1. The description of AvE lacks suffcient detail given it is the only baseline and addresses the same problem. Why does AvE struggle in more complex environments? How does it differ from ESR? \n\nMinor: \n\n1. The acronym ESR is not defined in the main text and is only defined in the caption of Algorithm 1. \n\nReferences:\n\n1. Carroll, Micah, et al. \"On the utility of learning about humans for human-ai coordination.\" Advances in neural information processing systems 32 (2019).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThanks for the review, and the suggestions for improving the paper. As suggested by the reviewer, we have evaluated the proposed method on a number of additional tasks, and compared with a number of additional baselines. **Together with the responses below, does this fully address the reviewer's concerns? About the paper?**\n\nKind regards,\n\nThe authors.\n\n\n> Evaluating on additional Overcooked environments\n\n\nTo expand our experimental results, we have run an additional experiment on the “asymmetric advantage” layout in Overcooked (see Table 1 in the attached pdf). We initially excluded this setting due to a lack of good models for human collaboration to evaluate on: policies that imitate expert data and/or use deep RL with self-play struggle to robustly learn collaborative behavior without additional structure. Our new result shows ESR can outperform baselines when playing with a heuristic-planning human model based on Carroll et al. (2019). \n\n> While the method addresses the problem of coordination without knowledge of the human's goal, the experiments demonstrating this are contrived. In Overcooked, the goal is fixed per task setting and is serving a dish.\n\nWe will add discussion of more complex settings that would be enabled by our method to the future work section. Examples of realistic settings where empowerment objectives could be effective include copilot-style assistants (Dohmke, 2022) and biomedical devices that improve human agency (Bryan et al., 2023; Pan et al., 2024). We note that the Overcooked environment has been used as a primary evaluation for human-AI collaboration in numerous works (Carroll et al., 2019; Hong et al., 2023; Knott et al., 2021; Laidlaw & Dragan, 2022; Lauffer et al., 2023; Strouse et al., 2021). \n\n\n> The paper does not show results in a setting where the goals and preferences of the human are complex to communicate.\n\n\nThis is a good point, the true reward of the Overcooked setting is easily communicated. However, our focus is on learning a collaborative policy without explicitly communicating the true reward, or the method of collaboration, which is more difficult to express.\n\n\n> The description of AvE lacks sufficient detail given it is the only baseline and addresses the same problem. Why does AvE struggle in more complex environments? How does it differ from ESR?\n\n\nThe key limitation of AvE is it tries to approximate the empowerment quantity which they define as $\\max_\\pi I(s^+;a_t\\mid s_t)$ through random rollouts. This is completely intractable in high-dimensional environments (as our larger experiments show), where random rollouts don’t produce meaningful behaviors. \n\nAdditionally, since the rollouts select actions uniformly at random, even in the limit of infinite computation, the AvE objective doesn’t actually compute empowerment with respect to meaningful human behaviors (either under their definition or our definition). Intuitively, imagine a human performing some task in a room with thin walls. The AvE objective would incentivize the robot knocking down the walls to maximize the human’s ability to leave the house, even though they have no desire to do so. Our ESR objective would help them with the task at hand.\n\n\n\n> The acronym ESR is not defined in the main text and is only defined in the caption of Algorithm 1.\n\nWe have fixed this.\n\n\n> L160 refers to \"the CIRL setting\", but CIRL is never defined. What does CIRL stand for?\n\nCIRL refers to the cooperative inverse reinforcement learning setup from Hadfield-Menell et al. (2016), in which an assistant cooperates with a human without knowing the true human reward.\n\n\n> As stated in the limitations, ESR assumes access to the human actions to learn the empowerment reward. Can ESR work with noisy predictions of human actions? \n\nIn general, the ESR objective under an arbitrary (state-independent) noise model becomes a lower bound on the “true” ESR empowerment (applying the data processing inequality to the Markov chain $\\hat{a}\\to a\\to s^+$). This suggests that ESR will be conservative in the presence of noisy action observations.\n\n\n> While ESR outperforms AvE, it greatly underperforms baselines from population-based training in [1].\n\nMethods that assume access to the true reward (i.e., Figure 4 of the Overcooked paper (Carroll et al., 2019)) will in general perform better than methods such as ESR that do not. \n\n> Is the assumption that the agents share the same state space central to the empowerment learning objective? Is it possible to implement ESR if both agents operate from separate egocentric views?\n\nWhile we do make this assumption in our formulation, in many cases it may be possible to use the same algorithm in partially observed settings. If the human has partial observability, this corresponds to maximizing empowerment of a suboptimal human (it throws away information about the state in its observation), which will often nevertheless be desirable. If the agent has partial observability, the empowerment objective will only increase the human’s influence over the parts of the environment the agent can observe. In many cases this may be fine, but in environments where the agent can restrict the information in its own observations, care must be taken to mitigate reward hacking (Amodei et al., 2016)—in this case, situations where the agent can choose to look at only the areas the human has influence over. We will add discussion on these points to the limitations and future work section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0z5dH67He1", "reviewer": "Reviewer_drkz", "summary": "The authors propose a new training objective that motivates the agent to assist humans by maximizing their empowerment, agnostic of their rewards. The proposed empowerment objective is derived from mutual information between human actions and future states, which is estimated via contrastive representation learning and can be formulated as an RL reward. The authors empirically show that this training objective improves performances in a grid world game and the overcooked game over existing baselines.", "questions": "Figure 2 confuses me. I would appreciate a better explanation of what each of these images is referring to.", "limitations": "The authors have addressed the limitations and potential negative societal impact.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The writing of the paper is clear and easy to follow.\n- The proposed method is novel to my knowledge.\n- The authors have shown theoretical insights into their definition of empowerment to a certain degree.", "weaknesses": "- My main concern is the lack of baselines. The only baselines that the author chose to compare are AvE, which is another empowerment-based method that this work is based on, and random. From the results, AvE shows to be weaker even than random in most of the scenarios, which makes it less appealing as a baseline. At least for the Overcooked domain, there are plenty of baselines in [1] to choose from (or at least argue why they are not chosen).\n- Following the previous point, even though there may not be enough empowerment-based methods to compare, the authors can still provide ablation studies to justify the design choices. As of now, the empirical result feels thin to me.\n- The authors could provide more insights in the experiment section. For example, how well does the proposed method estimate empowerment? Or qualitatively,  what does the agent do to increase empowerment?  More analysis can help us better understand the method.\n\n[1] Carroll, Micah et al. “On the Utility of Learning about Humans for Human-AI Coordination.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThanks for the detailed review and suggestions for improving the paper. To address the main concern about baselines, we have added two additional baselines (see below). We have also run additional ablation experiments, and tried to address the other concerns in the discussion below. **Does this fully address the reviewer's concerns?**\n\nKind regards,\n\nThe authors.\n\n\n> My main concern is the lack of baselines.\n\nWe have added a goal inference baseline based on Du et al. (2021) to the obstacle environment and a reward inference baseline using IQ-Learn (Garg et al., 2021) for both the obstacle environments and the overcooked settings. Our method outperforms both of these baselines across the environments studied. These experiments are included in the attached PDF.\n\nRegarding the baselines from Carrol et al. (2019) ([1] here): all of these require the true human reward. In contrast, our method does not assume access to the ground truth reward, and we aim to learn a collaborative policy through our empowerment objective.\n\n> provide ablation studies to justify the design choices.\n\nAs suggested by the reviewer, we have run additional ablation experiments, studying the effective of the contrastive objective and the empowerment parametrization (see Fig. 1 in the rebuttal PDF). \n\n> qualitative analysis \n\nWe have added additional qualitative results in the PDF.\nIn the obstacle environment ESR learns to remove obstacles along the path the human is moving and in the Overcooked setting it learns collaborative behaviors like moving the plate to places the human can reach and placing onions in the pot. We will revise the paper with visualizations of some of these behaviors.\n\n\n> Figure 2 confuses me.\n\nWe will clarify this figure caption in the revised paper. The goal of this figure is to relate the empowerment objective (Eq. 1) to the analysis of skill discovery (Lemma 2). On the left (a), we visualize how pairs of interacting policies induce a distribution over the state space, seen here as points on an $|\\mathcal{S}|$-simplex. In the center (b), the orange polygon depicts the set of state distributions that the human's policy can attain when working with a fixed robot assistant. The black lines correspond to Eq 3 (Lemma 1), which say that empowerment relates to the ``diameter'' of this polygon.  To the right (c), we show how our empowerment objective corresponds to maximizing the size of this polytope, i.e., maximizing the human’s ability to control the distribution over state distributions in the environment. We will revise the figure to clarify.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SLy1Bzvp90", "reviewer": "Reviewer_MFVx", "summary": "This paper introduces a method for assistance via empowerment based on contrastive successor representations, while also introducing a number of theoretical results about the relationship between assistive empowerment (understood as maximizing the mutual information between human actions and future states) and assistive reward maximization in information geometric terms. The proposed method, empowerment with successor representations (ESR) is shown in experiments to outperform a baseline method, AvE, which estimates empowerment using Monte Carlo rollouts. The experiments show that ESR scales to more complex problems than AvE, including environments with image based observations.", "questions": "POST-REBUTTAL: In light of the new experimental results comparing ESR to a reward inference baseline, I have raised my score to a 5. However, there remain substantial improvements to the presentation and theory-to-algorithm connection that should be made.\n\n===\n\nQuestions about empowerment definition:\n\n- Equation 1: What is the expectation in Eq (1) taken over? Is it over $\\pi_H , \\pi_R$? Is $s^+$ sampled $K$ steps into the future starting from each $t$? Or starting from $t = 0$?\n\nQuestions about information geometry:\n\n- Figure 2: What exactly is the 'center of the polytope\" here?\n- Lemma 1 and Lemma 2: What exactly is the skill $z$, and how is it related to human actions $a^H$?\n- Relatedly, how is the mutual information $I(s^+; z)$ related to $I(s^+; a^H_t | s_t)$?\n- In Lemma 2, what does it mean for a human to \"adapt to a reward function\"? If I'm understanding Eysenbach et al (2021) correctly, is the idea here that the human is modeled as starting from prior state distribution $\\rho(s)$, then adapting to $\\rho*(s)$ by learning to better optimize for the reward function (by learning new skills)?\n- In Lemma 2, I can see that the assistant maximizing the value of $I(s^+; z)$ leads to lower (regularized) regret for the human. But does maximizing the discounted sum of $I(s^+; a^H_t | s_t)$ also lead to lower regret for the human? It seems like this result isn't shown, and so Lemma 2 doesn't actually apply to the notion of empowerment used in the paper.\n- Line 159: \"We can view our objective as a generalization of the assistance problem beyond the CIRL setting\" --- I would be careful about making this claim. The assistance game setting is a (cooperative) Markov game, and the solution concepts in that setting are either optimal policies (Hadfield-Menell et al, 2016) or pragmatic-pedagogical equilibria (Fisac et al, 2017). In contrast, Lemma 2 only shows that maximizing mutual information corresponds to minimizing regularized regret --- which is not the same as finding the optimal joint policy, or finding a pragmatic-pedagogical equilibrium.\n\nQuestions about contrastive representation learning:\n\n- Line 197: What is $g$? A future state? Why not use $s^+$ as before?\n- Line 198: Why do $\\phi$ and $\\phi'$ each suddenly lose one argument?\n- Line 201: Please provide some derivation or explanation as to why these representations would encode the stated probability ratios upon convergence.\n- Equations 8 and 9: Why are the conditional probabilities not conditioned on $a^R_t$?\n\nQuestions about experiments:\n\n- What embedding functions or neural network did you use to learn the successor features in each benchmark?\n- How does ESR compare against goal inference baselines, e.g. those used in Du et al (2020) or Laidlaw et al (2024)?\n\nMinor Comments:\n\n- There are a number of typos here and there (\"gradefully\", \"belore\") which should be fixed.\n- Line 256: \"While much of the amazing work\" --- too subjective for an academic paper.", "limitations": "The authors adequately discuss the technical limitations of their approach. They also have appropriately noted the risks of focusing solely on empowerment for assistance, as this might empower actors who already unjustly have too much power.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper introduces a new method, ESR, for training agents to assist users by maximizing empowerment, which does not require the agent to infer human goals or preferences (as in methods based on inverse planning or inverse reinforcement learning). Unlike previous methods for assistance via empowerment, which make use of Monte Carlo rollouts to estimate variance, ESR's method for estimating empowerment is more scalable, drawing upon ideas in contrastive representation learning to effectively estimate the mutual information between human actions and future states. This allows ESR to be applied to more complex problems, increasing the applicability of assistance via empowerment to more contexts. The underlying ideas behind this method are interesting, and the method appears to be effective (at least with respect to the AvE baseline). As such, I believe the paper will be of some interest to others working on AI assistance and human-AI alignment.", "weaknesses": "While the ideas behind this paper are interesting, and the method seems to work reasonably well, the presentation of the framework could be significantly improved. The empirical evaluations could also be made more rigorous by comparing against non empowerment-based baselines.\n\n**Presentation**\n\nThis paper was hard to understand on its own, even after reading the Appendix. I had to read Eysenbach (2021) to really understand the theoretical results, and van den Oord (2019) to understand how the proposed method worked. I think a well-presented paper wouldn't have had that issue.\n\nIn the information geometry section, important quantities such as the skill variable $z$ and state occupancy measure $\\rho(s)$ are not properly defined. This makes it hard to understand all the different state occupancy measures $\\rho(z)$, $\\rho^+(z)$, $\\rho^*(z)$ that are introduced, and what exactly the prior over states means (or why the human would be able to choose this prior). As a result, it's hard to evaluate the soundness of the arguments (without, e.g., reading Eysenbach (2021)), or whether the assumptions about the human policy underlying Lemma 2 are reasonable ones. It's also not explained how the mutual information $I(s^+; z)$. between future states $s^+$ and human skills $z$ is related to the definition of empowerment in Equation (1), which involves the (conditional) mutual information $I(s^+; a^H_t | s_t)$ between future states and human actions $a^H_t$. Since this is not explained, it's hard to see how Section 3.2 and 3.3 are relevant for actually maximizing empowerment as it's defined in Section 3.1.\n\nIn section 4, where the ESR algorithm is introduced, it's also not explained how or why the contrastive representation learning objective in Eq. (7) should lead to right density ratios to be estimated upon training convergence. One has to read van den Oord (2019) to understand why, so this section is not really accessible to readers not already familiar with contrastive learning methods. Notation is also not clearly defined -- for example, why does $\\phi$ switch from being a 3-argument function in Line 191 to becoming 2 argument function in Line 198? I have the inkling that this has to do with marginalizing out the assistant's action $a^R$, but this is not explained. I'm also confused why $a^R$ needs to be part of the successor representation at all. By including $a^R$, won't the resulting mutual information you're estimating end up being $I(s^+; a^H_t | s_t, a^R_t )$, which is conditional on *both* the current state $s_t$ and the assistant's action $a^R_t$?\n\nPerhaps one reason why this paper ends up being hard to follow is because it's trying to do too much by both introducing the results in Sections 3.2-3.3, while also introducing a (seemingly unrelated) method for empowerment maximization in Section 4. As it stands, these two parts of the paper feel quite disjoint to me, and it's not obvious how they form a cohesive whole. It might have been better to focus on just one or the other -- e.g. on just explaining and carefully justifying the method in Section 4 -- so that everything is more understandable and cohesive.\n\n**Evaluation**\n\nEven though one of the main selling points of assistance via empowerment is that it does not require performing goal inference (which can be hard to scale when the goal space is large), the experiments do not compare ESR against any goal inference methods. This is in contrast to the original AvE paper by Du et al (2020), which does conduct fairly thorough comparisons against goal inference baselines. As a result, it's hard to evaluate exactly how valuable ESR is, and whether one should prefer it over goal inference as an assistance method. This should be addressed in future versions of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for the detailed review, and for the suggestions for improving the paper. It seems like the reviewer's main concern is about baselines and presentation. We have attempted to address these concerns by adding XX additional baselines, and by significantly revising the paper (incorporating the reviewer's suggestions). **Together with the discussion below, does this fully address the reviewer's concerns?** If not, we look forward to continuing the discussion!\n\nKind regards,\n\nThe authors.\n\n> How does ESR compare against goal inference baselines, e.g. those used in Du et al (2020) or Laidlaw et al (2024)?\n\nWe have added goal inference baselines for the obstacle environment, with our ESR method achieving the best performance (see attached PDF). In contrast with Du et al (2020), which assumes a model-based, almost “oracle” goal inference, we implement a model-free baseline for fair comparison. We have revised the paper to describe this.\n\nIn the Overcooked environments, there is no clear notion of a goal (the environment returns roughly to the same state it started in after the soup is made and delivered).  An advantage of our method over goal inference is that it is well-defined in such settings. \n\n> Equation 1: What is the expectation in Eq (1) taken over? Is it over $\\pi_H, \\pi_R$ ? Is $s^{+}$sampled $K$ steps into the future starting from each $t$ ? Or starting from $t=0$?\n\nThis expectation is taken over a trajectory $(s_0, s_1, \\cdots)$ sampled when the human ($\\pi_H$) and robot ($\\pi_R$) interact in the environment. Thus, the random variable $s_t$ corresponds to time step $t$ in an episode. Random variable $s^+$ is sampled from the discounted state occupancy measure conditioned on $s_t$; in other words, the sampling procedure can be written as $K \\sim Geom(1 - \\gamma)$ and setting $s^+ = s_{t+K}$. We have revised the paper to clarify this.\n\n> important quantities such as the skill variable z and state occupancy measure ρ(s) are not properly defined.\n\nWe will revise the paper to more clearly connect the notation used for the analysis to the main text.\n\n> how the mutual information  I(s+;z) . between future states s+ and human skills z is related to the definition of empowerment in Equation (1), which involves the (conditional) mutual information I(s+;atH|st) between futurestates and human actions atH In section 4, where the ESR algorithm is introduced, it's also not explained how or why the contrastive representation learning objective in Eq. (7) should lead to right density ratios to be estimated upon training convergence. One has to read van den Oord (2019) to understand why, so this section is not really accessible to readers not already familiar with contrastive learning methods. \n\nThank you for pointing this out! We agree that it is difficult to understand the mathematical details without this background. In our revision, we will add a derivation of the solution to our symmetrized infoNCE objective in Appendix D, which follows directly from the analysis by Poole et al. (2019) when adapted to the symmetric version of the loss as in Radford et al. (2021) and Eysenbach et al. (2024).\n\n> why aR needs to be part of the successor representation at all. By including aR, won't the resulting mutual information you're estimating end up being I(s+;atH|st,atR), which is conditional on both the current state st and the assistant's action atR ?\n\nWe found conditioning on a_R as well helped stabilize training. A challenge with training an algorithm like ESR is that as $\\pi_R$ changes during training, it affects the successor features and the computed empowerment reward. By conditioning the representation on the current policy’s actions, we improve the ability of the empowerment bonus to “keep up” with the policy during training. We will revise the main text to indicate this more clearly.\n\n> Lemma 1 and Lemma 2: What exactly is the skill $z$, and how is it related to human actions $a^H$ ?\n\nA ``skill'' is purely a mathematical construction used in our analysis. The analysis focuses on the distribution over states visited by the human and the robot. Our analysis represents the robot's distribution over states $\\rho(s)$ as a mixture, $\\rho(s) = \\sum_z \\rho(s \\miz s) p(z)$. We have revised the paper to clarify this.\n\n> Relatedly, how is the mutual information $I(s^+; z)$ related to $I(s^+; a_t^H \\mid s_t)$?\n\nAll instances of random variable $z$ should be replaced by $a^H$; we apologize for any confusion caused by this typo, which was caused by different prior work using different notation. Our theoretical analysis looks at the mutual information $I(s^+; a_t^H)$, which measures the ability of the human to effect change in their environment. Our practical method says that we should maximize this objective at all states: $I(s^+; a_t^H \\mid s_t)$ looks at the ability to effect change starting at state $s_t$, and our practical algorithm aims to maximize this objective over all visited states $s_t$ (see Eq. 1).\n\n> In Lemma 2, what does it mean for a human to \"adapt to a reward function\"? If I'm understanding Eysenbach et al (2021) correctly, is the idea here that the human is modeled as starting from prior state distribution $\\rho(s)$, then adapting to $\\rho *(s)$ by learning to better optimize for the reward function (by learning new skills)?\n\nYes, that is correct. In the context of this paper, we look at a restricted set of skills: those defined by (open-loop) sequences of actions, $a_t^H$. This is why we use $a_t^H$ in our mutual information objective, rather than the letter $z$ used in (Eysenbach et al 2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fnIcshe7Tq", "reviewer": "Reviewer_KT7u", "summary": "This paper presents Empowerment via Successor Representations or ESR, a technique that builds an assistive agent that maximizes the human collaborator's ability to influence the world. The key motivation the authors provide for building an assistive agent that seeks to empower the human rather than explicitly provide support by inferring the human collaborator's reward function is that inferring such a reward function can be challenging. The authors present a formulation for empowerment and connect empowerment maximization to reward maximization in Section 3. The authors then present an implicit approach to estimate empowerment via several learned representations and present loss functions for how these representations can be inferred and utilized. Finally, the authors present a set of experiments displaying that this technique can outperform a related prior work (AvE) as well as a random baseline.", "questions": "Other Questions/Recommendations:\n- Could you please comment on the weaknesses above?\n- Could you comment on the feasibility of this approach with actual humans? How many samples would be needed to infer semi-accurate representations and create agents that could collaborate well? Further, if the answer is learning these representation a priori, could you comment on how this framework would work in cases where you do not have access to a high-fidelity simulator.\n- There is some notation that is unclear and grammar issues/typos. I would recommend checking to ensure that all variables are defined, which will help improve clarity.\n- Figure Descriptions could be improved", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Strengths\n+ The presented formulation is very interesting and shows promise. To the best of my knowledge, the utilization of such representations to estimate empowerment is new.\n+ Generally, the results are impressive. It is clear that ESR outperforms the prior technique by large margins in the chosen domains.", "weaknesses": "Weaknesses\n- Paper's claims should be better grounded:\n1. For the proof in Section 3.3, the conclusion is that an assistive agent maximizing the mutual information between a future state and a chosen skill by the human minimizes the worst-case regret incurred by the human. Is this only true for a two-agent scenario and full observability? Does the proof have any assumptions for the robot and human maintaining the same representation and action space?\n2. The paper mentions that inferring a human's reward function can be error-prone and challenging. However, I would think inference of several representations (especially online), could lead to similar outcomes (poor consequences etc.) It isn't clear why inferring these approximate representations and seeking to maximize empowerment would work better than reward inference. I'm specifically thinking about the cases mentioned in the introduction, where humans have changing preferences, act suboptimally, etc. \n3. The term high-dimensional is used in the introduction but the chosen domains seem to be relatively low-dimensional compared to other collaborative testbeds encountered in robotics (see [1]) and human-AI collaboration (see [2]). Could you clarify this term?\n- Evaluation is limited: \n1. As this work aims to develop assistive agents that work with humans, it should compare with actual humans in a user study to validate the synthetic findings. \n2. The evaluation is only conducted with respect to one baseline (AvE) and a random agent. The author's key rationale behind this technique was assisting humans without inferring reward functions, so it would be beneficial to test against a framework that actively infers the user's reward function. \n- Some claims are not well-explained.\n1. The introduction notes that AvE does not scale to high-dimensional settings but does not make it clear why this framework failed to scale.\n2. The authors show and mention AvE underperforms a random agent but does not mention why. Could you provide further details? Also, could the authors comment on why ESR was able to outperform AvE by such a wide margin and note any qualitative findings about the learned collaborative behavior?\n\n\n[1] Liu, Puze, et al. \"Safe reinforcement learning of dynamic high-dimensional robotic tasks: navigation, manipulation, interaction.\" 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.\n\n[2] Paleja, Rohan, et al. \"The utility of explainable ai in ad hoc human-machine teaming.\" Advances in neural information processing systems 34 (2021): 610-623.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for the detailed review, and for all the suggestions for improving the paper.\nBased on the reviewer's feedback, we have run experiments with additional baselines and clarified several parts of the writing. \n**Together with the discussion below, does this fully address the reviewer's concerns with the paper?** We look forward to continuing the discussion!\n\nKind regards,\n\nThe authors\n\n> It isn't clear why inferring these approximate representations and seeking to maximize empowerment would work better than reward inference.\n> it would be beneficial to test against a framework that actively infers the user's reward function.\n\nTo answer this question, we ran an additional experiment comparing our method (which estimates representations) to a reward learning approach that uses IQ-Learn (Garg et al., 2021) as an IRL algorithm to infer human rewards and then use these to train the assistant, as in the CiRL (Hadfield-Menell et al., 2016) framework. The results (rebuttal PDF Figs. 1 and 3) show that our approach outperforms this baseline.\n\n> Is this only true for a two-agent scenario and full observability?\n\nYes. We have revised the paper to add these assumptions. We believe that handling more than two agents should be feasible (in future work) but handling partial observability may be more challenging. Under certain assumptions (e.g., state-independent noise in observing human actions or bounded divergence between the agent’s belief and the true environment) we can reason about these challenges. But without any assumptions pathological MDPs can be constructed.\n\n> Does the proof have any assumptions for the robot and human maintaining the same representation and action space?\n\nNo, they do not need to have the same representations or action space. Our implementation re-uses one representation ($\\psi(g)$) purely for computationally efficiency, but this can be removed if an algorithm user wanted to use entirely disjoint representations.\n\n> Clarify the meaning of high-dimensional\n\nOur states for both environments are images: 5x5 with 3 feature channels for the obstacle gridworld and 4x5/5x5 with 26 feature channels for the Overcooked environments.\n\n\n> Why does AvE perform so poorly on these experiments?\n\nThe key limitation of AvE is it tries to approximate the empowerment quantity which they define as $\\max_\\pi I(s^+;a_t\\mid s_t)$ through random rollouts. This is completely intractable in high-dimensional environments (as our larger experiments show), where random rollouts don’t produce meaningful behaviors. \n\nAdditionally, since the rollouts select actions uniformly at random, even in the limit of infinite computation, the AvE objective doesn’t actually compute empowerment with respect to meaningful human behaviors (either under their definition or our definition). Intuitively, imagine a human performing some task in a room with thin walls. The AvE objective would incentivize the robot knocking down the walls to maximize the human’s ability to leave the house, even though they have no desire to do so. Our ESR objective would help them with the task at hand.\n\n> why ESR was able to outperform AvE by such a wide margin and note any qualitative findings about the learned collaborative behavior?\n\nFor the reasons noted above, AvE is unable to perform well on the more complex tasks studied, while ESR can scalably empower the human. Qualitatively, in the obstacle environment ESR learns to remove obstacles along the path the human is moving and in the Overcooked setting it learns collaborative behaviors like moving the plate to places the human can reach and placing onions in the pot. We will revise the paper with sketches of some of these behaviors.\n\n> Could you comment on the feasibility of this approach with actual humans?\n\nThis paper is primarily mathematical and algorithmic and nature, and so we'll provide an mathematical/algorithmic answer: Eq. (6) says that we can measure the degree of (minimax) compatibility via the mutual information. We have run an additional experiment to plot this mutual information throughout training (see Fix X in rebuttal PDF). Visualizing the learned agent, we see that the agent does indeed become more helpful as this mutual information increases.\n\nOf course, the gold standard in human-AI interaction is human user studies, which are beyond the scope of this mathematical/algorithmic paper.\n\n\n> checking to ensure that all variables are defined, which will help improve clarity.\n\nThanks for the suggestion – we have done this with our local working copy of the paper.\n\n> Figure Descriptions could be improved\n\nWe have clarified the figure descriptions, incorporating the feedback from the other reviewers as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their feedback. Reviewers mentioned concerns about baselines and presentation, which we have responded to in detail below. Based on this feedback, we have run additional ablations and baselines for our method and conducted additional qualitative analysis (see attached PDF).

Meta-review: This paper initially received a two borderline accepts (FYis, drkz), an accept (PN73), a borderline reject (MFVx), and a reject (KT7u). All reviewers appreciated the novelty of building assistive agents that improve human empowerment, particularly in higher dimensional spaces, and the theoretical contributions supporting this. Beyond requesting clarifications, the reviewers raised concerns about baselines and the sufficiency of empirical results (including lack of human evaluation). 

In the discussion phase, the authors addressed most of these concerns by adding new baselines to their benchmarks, presenting new results on the asymmetric advantage layout of overcooked, including additional ablations for design choices, and clarifying the assumptions and scope of their work. Based on this, reviewers FYis, MFVx, and KT7u increased overall ratings, and drkz expressed they are more inclined to accept the paper. 

The AC concurs with the reviewers on the novelty of this work and recommends acceptance. While human study & empirical results of human compatibility would strengthen this work, the paper’s current theoretical and empirical contributions hold significant value. The authors are encouraged to improve the paper’s readability for the final version with a systematic introduction of new terminology (PN73), edits to figures (drkz), and rigorous mathematical formalization (MFVx) -- based on the constructive discussion between the authors and the reviewers.
Decision: Accept (poster)