Benefits of Assistance over Reward Learning
Authors: Rohin Shah
Pedro Freire
Neel Alex
Rachel Freedman
Dmitrii Krasheninnikov
Lawrence Chan
Michael D Dennis
Pieter Abbeel
Anca Dragan
Stuart Russell
Venue:  (2021)
Keywords: assistance
reward learning
preference learning
active learning

Abstract: Much recent work has focused on how an agent can learn what to do from human feedback, leading to two major paradigms. The first paradigm is reward learning, in which the agent learns a reward model through human feedback that is provided externally from the environment. The second is assistance, in which the human is modeled as a part of the environment, and the true reward function is modeled as a latent variable in the environment that the agent may make inferences about. The key difference between the two paradigms is that in the reward learning paradigm, by construction there is a separation between reward learning and control using the learned reward. In contrast, in assistance these functions are performed as needed by a single policy. By merging reward learning and control, assistive agents can reason about the impact of control actions on reward learning, leading to several advantages over agents based on reward learning. We illustrate these advantages in simple environments by showing desirable qualitative behaviors of assistive agents that cannot be found by agents based on reward learning.

Under review as a conference paper at ICLR 2021
BENEFITS OF ASSISTANCE OVER REWARD LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Much recent work has focused on how an agent can learn what to do from human feedback, leading to two major paradigms. The first paradigm is reward learning, in which the agent learns a reward model through human feedback that is provided externally from the environment. The second is assistance, in which the human is modeled as a part of the environment, and the true reward function is modeled as a latent variable in the environment that the agent may make inferences about. The key difference between the two paradigms is that in the reward learning paradigm, by construction there a separation between reward learning and control using the learned reward. In contrast, in assistance these functions are performed as needed by a single policy. By merging reward learning and control, assistive agents can reason about the impact of control actions on reward learning, leading to several advantages over agents based on reward learning. We illustrate these advantages simple environments by showing desirable qualitative behaviors of assistive agents that cannot be found by agents based on reward learning.
INTRODUCTION
Traditional computer programs are instructions on how to perform particular task. However, we do not know how mechanically perform more challenging tasks like translation. The field of artificial intelligence raises the level of abstraction so that we simply specify what the task is, and let the machine to figure out how to do it.
As task complexity increases, even specifying the task becomes difficult. Several criteria that we might have thought were part of a specification of fairness turn out to be provably impossible to simultancously satisfy (Kleinberg et al., 2016 Chouldechova 2017: Corbett-Davice Reinforcement learning agents often "game" their reward function by finding solutions that technically achieve high reward without doing what the designer intended (Lehman et al.) 2018 Krakovna 2018
al. 2017).
Clark & Amodei 2016). In complex environments, we need to specify what not to change (McCarthy & Hayes 1981;; failure do so can lead negative side effects (Amodei et 2016). Powerful agents with poor specifications may pursue instrumental subgoals (Bostrom) 1014/unooo 2008 such as resisting shutdown and accumulating resources and power (Turner (019))
A natural solution is to once again raise the level of abstraction, and create an agent that is uncertain about the objective and infers from human feedback, rather than directly specifying some particular task(s). Rather than using the current model of intelligent agents optimizing for their objectives, we would now have beneficial agents optimizing for our objectives (Russell 2019).
Reward learning (Leike et al. 2018; Jeon et al.) 2020 Christiano et al.) 2017] Ziebart et al., 2010) attempts to instantiate this by learning a reward model from human feedback, and then using a control algorithm to optimize the learned reward. Crucially, the control algorithm does not reason about the effects of the chosen actions on the reward learning process, which is external to the environment.
In contrast, in the assistance paradigm (Hadfield-Menell et al., 2016 Fern et al. 2014), the human H is modeled as part of the environment and as having some latent goal that the agent R (for robot) does not know. R's goal is to maximize this (unknown) human goal. In this formulation, R must balance between actions that help learn about the unknown goal, and control actions that lead to high reward.
Our key insight is that by integrating reward learning and control modules, assistive agents can take into account the reward learning process when selecting actions. This gives assistive agents a significant advantage over reward learning agents, which cannot perform similar reasoning.

---

Under review as a conference paper at ICLR 2021
All pies need M Jet me make it
It use the L make there won't be any left
for and I'll wait for more information.
M Making robust plans
Preserving option value when possible
hal the reward
won't 't find out which pie is preferable before Afice gets very hungry, make
Leaming about reward
Guessing when feedback is unavailable
Figure 1: R must cook pie for H, by placing flour on the plate to make the pie dough, filling it with either Apple, Blueberry, or Cherry filling. and finally baking However, R does know which filling H prefers, and H is not available for questions since she is doing something else. What should R do in this situation? On the right, we show what qualitative reasoning we might want to use to handle the situation.
The goal of this paper is to clarify and illustrate this advantage. We first precisely characterize the differences between reward learning and assistance, by showing that two phase, communicative assistance is equivalent to reward learning (Section 3). We then give qualitative examples of desirable behaviors that only be expressed once these restrictions are lifted, and thus only exhibited by assistive agents (Section 4
Consider for example the kitchen environment illustrated in Figure I in which R must bake pie for H. R uncertain about which type of pie H prefers to have, and currently H is at work and cannot answer R's questions. An assistive R can make the pie crust, but wait ask // about her preferences over the filling (Section 4.1). R may never clarify all of H's preferences: for example, I only needs know how dispose of food if it turns out that the ingredients have gone bad (Section 4.2). If H will help with making the pie. R can allow H to disambiguate her desired pie by watching what filling she chooses (Section 4.3). Vanilla reward learning agents do not show these behaviors.
We do not mean to suggest that all work on reward learning should cease and only research on assistive agents should be pursued. Amongst other limitations, assistive agents are very computationally complex. Our goal is simply to clarify what qualitative benefits an assistive formulation could theoretically provide. Further research is needed to develop efficient algorithms that can capture these benefits. Such algorithms may look like algorithms designed to solve assistance problems as we have formalized them here, but they may also look like modified variants of reward learning. where the modifications designed to provide the qualitative benefits identify.
2
BACKGROUND AND RELATED WORK
We introduce the key ideas behind reward learning and assistance. X denotes a sequence of X. We use parametric specifications for case of exposition, but our results apply more generally.
2.1 POMDPS
partially observable Markov decision process (POMDP) M = (S, A, Po. 7) con- sists of a finite state space S, a finite action space A, finite observation space Ω an observation function S + A(l) (where A(X) is the of probability distributions over X), transition function T : S x A + (SS) reward function : S x A X S + IR an initial state distribution Po : (S), and a discount rate 7 E (0, 1). We will write of to signify the fth observation (s:). A solution to the POMDP is given by a policy π (O x A)* x O + ((A) that maximizes the expected Eag Paan-Pa.rseee x loo -()).))
sum of rewards ER(x) x))
De 1 ( ar st-l)].
2.2 REWARD LEARNING
We consider variants of reward learning: non-active reward learning, in which R must infer the reward by observing H's behavior, and active reward learning, in which may choose particular questions to ask H in order to get particular feedback.
non-active reward learning problem P = (Mir, C, (9, ro. Po), ,..k), k) contains a POMDP without reward M'r (S, R AR OR, T. Po, 7), and instead R has access to parameterized
2

---

Under review as a conference paper at ICLR 2021
reward space (0,ra, ra, Pe). R is able to learn about 8* by observing H make io different choices each chosen from a set of potential choices C. In order for R to learn from the human's 's choices, it also assumes access to the human decision function x (c 0) that determines how the human makes choices for different possible reward functions rg. Common decision functions include perfect
optimality (Ng & | 2000) and Boltzmann rationality (Ziebart al., 2010). There many types of choices (Jeon et al.] 2020), including demonstrations (Argall et al. 20099 Ng & Russell 2000; Ziebart et al., | 2010; Fu et al. 2017 Gao et al. 2012), comparisons (Zhang et al. 2017; Wirth
et al.,
et al. 2017; Sadigh et 2017), corrections (Bajesy et al. (017)) 7) the state of
the world (Shah et al. 2019). proxy rewards (Hadfield Menell et (017b), natural language (Fu
et al. 2019), etc.
policy decision function (coos=11) produces a policy R after observing H's choices. A solution is policy decision function f that maximizes expected reward -.t= Since H's choices co.k-1 do not affect the state of the environment that R is acting in, this equivalent to choosing π that maximizes expected reward given the posterior over reward functions, that is P(0 [ER(T")]
An active reward learning problem P (Mir. Q, (0, Pe), " k) adds the ability for R to ask H particular questions g E Q order to get more targeted feedback about 0. The human decision function π [cl q, 0) now depends on the question asked. A solution consists of a question policy TB(9: (: 90:1-1, G0s-1) 1) and a policy decision function f(qok 1, 1 that maximize expected
reward Egg..e 1 1 Poie 1
[ER( [f 10
Co:k-1 1)]]
A typical algorithm (Eric et al. 2008 Daniel et al. 2014 Maystre & Grossglauser 2017 Christiano
al. 2017 Sadigh al.] 2017 Zhang al. 2017: Wilde et al., 2020) will compute and ask g E Q
that maximizes an active learning criterion such as information gain (Biyik et al.] 2019) or volume removal (Sadigh al.) (017). Best results are achieved by selecting questions with the highest value of information (Cohn, Robert W. 2016; Zhang et al.) [2177] Mindermann et al., 2018 Wilde ct al. 2020), but these are usually much more computationally expensive. R then finds a policy that maximizes expected reward under the inferred distribution over 6, in order to approximately solve the original POMDP.
Note that a non-active reward learning problem is equivalent to an active reward learning problem with only one question, since having just a single question means that R has no choice in what feedback to get (see Appendix A. for proofs).
2.3 ASSISTANCE
The key idea of assistance is that helpful behaviors like reward learning are incentivized when R does not know the true reward r and can only learn about it by observing human behavior. So, we model the human H as part of the environment, leading a two-agent POMDP, and assume there is some true reward r that only H has access to, while the robot R only has access to a model relating r to H's behavior. Intuitively, as R acts in the environment, it will also observe H's behavior, which it can use to make inferences about the true reward.
Following Hadfield-Menel al. 2 we define an assistance game M tuple
M = (S, (A A)) (2 _ (0" O"), T, Ps. 7, (0, Pe))-
Here S is finite set of states, A a finite set of actions for H, f a finite set of observations for observation function for Il (respectively A", ? O for R). The
H. and O S + A(?") transition function T' S x A x A + A(S) gives the probability over next states given the current state and both actions. The initial state is sampled from Ps E ASS). θ is set of possible reward function parameters which parameterize a class of reward functions T9 : S x A x A R x S + R. and Po is the distribution from which 0 is sampled. 7 E (0, 1) is discount factor.
As with POMDPs, policies can depend on history. Both H and R are able to observe each other's actions, and on given timestep, R acts before H. We use 7. (ΩR A x A to denote
Relative to Haddield- Menell et al. (2116). our definition allows for partial observability and requires that the initial distribution over S and @ be independent. We also have и choose her action sequentially after R, rather than simultancously with R, in order to hetter parallel the reward learning setting.
3

---

Under review as a conference paper at ICLR 2021
R's observations until time t, and r. for H's observations: thus R's policy can be written as v²(a" RR - T(=)), while H's can be written as T " r" (a" a aa T/.1 0). Note that unlike H, R does not observe the reward parameter 8, and must infer 0 much like it does the hidden state.
A fully observable assistance game is one in which both H and R can observe the full state. In such cases, omit 2 22 O# and OR
Since we have not yet specified how H behaves, it is not clear what the agent should optimize for. Should it be playing Nash strategy optimal strategy pair of the game, and if so, which one? Should it use a non-equilibrium policy, since humans likely do not use equilibrium strategies? This key hyperparameter in assistance games, as determines the communication protocol for H and R. For maximum generality, we can equip the assistance game with a policy-conditione belief B 11tt + A(11") over T which specifies how the human responds to the agent's ''s choice of policy (Halpern & Pass 2018). The agent's s goal is to maximize expected reward given this belief.
Prior work on assistance games (Hadfield-Menell et al. 2016 Malik al.. 2018 Woodward et al.. 2019) focuses on finding optimal strategy pairs. This corresponds to a belief that H will know and per- fectly respond to II's policy (see Appendix However, our goal is to compare assistance to reward learning. Typical reward learning algorithms assume access to model of human decision-making: for example, H might be modeled as optimal (Ng & Russell, 2000) or Boltzmann-rations (Ziebart et al. 2010). As result, we also assume that we have access to model of human decision- makinn π Note that n depends on O: we are effectively assuming that we know how H chooses how to behave given particular reward ra. This assumption corresponds to the colicy-conditioned belief B(TR)(# [ = x = We define an assistance problem P as a pair (M, x where π H is human policy for the assistance game M.
Given an assistance problem, a robot policy RR induces a probability distribution over trajectories: (so, 0, n" m E [S A AR) We denote the support of this distribution by Traj( The expected reward of a robot policy for (M, x ")) is given by
00 Σ
n'ro (8,, af af sCHI)
ER((T
E sor-Ps 0~ Pa. (50,0,
A solution of (M, ) is a robot policy that maximizes expected reward: π " argmax E E(
2.3.1 SOLVING ASSISTANCE PROBLEMS
Once the n is given, H can be thought of as an aspect of the environment, and 0 can be thought of as particularly useful piece of information for estimating how good actions are. This suggests that we can reduce the assistance problem to an equivalent POMDP. Following Desai (2017), the key idea to embed /² the transition function T and embed in the state.
In theory. to embed potentially non-Markovian π If in T, we need to embed the entire history of the trajectory in the state, but this leads to extremely large POMDPs. In our experiments, we only consider Markovian human policies, for which we do not need to embed the full history, keeping the state space manageable. Thus, the policy can be written as -h (a" of aa. 0). To ensure that R must infer 0 from human behavior, in the original assistance game, the observation function does not reveal 0, but does reveal the previous human action "².
Proposition 1. Every assistance problem (M,TH)) can be reduced to an equivalent POMDP M'.
The full reduction and proof of equivalence is given in Appendix A.2
When M is fully observable, in the reduced POMDP 0 is the only part of the state not directly observable to the robot, making it an instance of a hidden-goal MDP (Fern et al.) 2014). For computational tractability, much of the work on hidden goals (Javdani et al. 2015 Fern et al. 2014) selects actions assuming that all goal ambiguity is resolved in one step. This effectively separates reward learning and control in the same way as typical reward learning algorithms, thus negating many of the benefits we highlight in this work. Intention- aware motion planning (Bandyopadhyay ett al., 2013) also embeds the human goal in the state in order to avoid collisions with humans during motion planning, but does not consider applications for assistance.
4

---

Under review as a conference paper at ICLR 2021
Macindoe et al. (2012) uses the formulation of POMDP with a hidden goal to produce assistive agent in a cops and robbers gridworld environment. Nikolaidis et al. (2015) assumes a dataset of joint numan-robot demonstrations, which they leverage to learn "types" of humans that can then be inferred online using a POMDP framework. This is similar to solving an assistance problem, where we think of the different values of f different "types" of humans. [Chen et al. (2018) uses an assistance-style framework in which the unknown parameter is the human's 's trust in the robot (rather than the reward 0). Woodward et al. 2019) uses deep reinforcement learning to solve an assistance game in which the team must collect either plums or lemons. To our knowledge, these are the only prior works that use an assistive formulation in a way that does not ignore the information -aateriigg aspect of actions. While these works typically focus on algorithms to solve assistance games, we instead focus on the qualitative benefits of using an assistance formulation.
Since we can reduce an assistance problem to regular POMDP, we can use any POMDP solver to find the optimal R In our examples for this paper, we use an exact solver when feasible, and point-based value iteration (PBVI) (Pineau et al. 2003) or deep reinforcement learning (DRL) when not. When using DRL, we require recurrent models, since the optimal policy can depend on history.
A common confusion is to ask how DRL can be used, given that it requires reward signal, but by assumption R does not know the reward function. This stems from a misunderstanding of what means for R "not to know" the reward function. When DRL is run, at the beginning of each episode, a specific value of 0 is sampled as part of the initial state. The learned policy π is not provided with 0: it can only
its observations of and human actions a and it is accurate to say that æ² "does not know" the reward function. However, the reward is calculated by the DRL algorithm, not by n², and the algorithm can and does use the sampled value of 0 for this computation. T can then implicitly learn the correlation between the actions al chosen by π and the high reward values that the DRL algorithm computes; this can be often be thought of as an implicit estimation of 0 in order to choose the right actions.
3 REWARD LEARNING AS TWO-PHASE COMMUNICATIVE ASSISTANCE
There are two key differences between reward learning and assistance. First, reward learning algorithms split reward learning and control into two separate phases, while assistance merges them into single phase. Second, in reward learning, the human's 's only role is to communicate reward information to the robot, while in assistance the human can help with the task. These two properties exactly characterize the difference between the two: reward learning problems and communicative assistance problems with two phases can be reduced to each other, in a very natural way.
A communicative assistance problem is one in which the transition function T and the reward function rg are independent of the choice of human action " and the human policy π (- o" aa .) independent of the observation o Thus, in communicative assistance problem, H's 's actions only serve to respond to R, and have no effects on the state or the reward (other than by influencing R). Such problems can be cast as instances of HOP-POMDPs (Rosenthal & Veloso 2011).
For the notion of two phases, we will also need to classify robot actions as communicative or not. We will assume that there is some distinguished action a khoop that "does nothing". Then, a robot action R is communicative if for any 8, a ,s' we have T(s' |s, a (R)) = T(s' s, ss goop)) and R(s, H _a 8') = R(s, al .aRoopp 8'). A robot action is physical if it is not communicative. (R,"))
Now consider communicative assistance problem (M, (, with noop action Koop and let the optimal robot policy be R Intuitively, we would like say that there is an initial communication phase in which the only thing that happens is that H responds to questions from R. and then a second action phase in which Hl does nothing and R acts. Formally, the assistance problem is two phase with actions at fact if it satisfies the following property:
Aa E A VT E T*))(
[VL < last R is communicative A V 2 lact
H = - koop))
Thus, in a two phase assistance problem, every trajectory from an optimal policy can be split into a "communication" phase where R. cannot act and an "action" phase where H cannot communicate.
Reducing reward learning to assistance. We can convert an active reward learning problem to a two-phase communicative assistance problem in an intuitive way: add to the set of robot actions,
5

---

Under review as a conference paper at ICLR 2021
make C' the set of human actions, add timestep counter to the state, and construct the reward such that an optimal policy must switch between the two phases after l questions. A non-active reward learning problem can first be converted to active reward learning problem.
Proposition 2. Every active reward learning problem (M, Q, C, (O, Pe), n", can be reduced to an equivalent two phase communicative assistance problem "M""". Corollary 3. Every non-active reward learning problem (M, C, (O, re. Pe), To),"", ,k) can be reduced an equivalent two phase communicative assistance problem (M', "
Reducing assistance to reward learning, The reduction from a two-phase communicative assis- tance problem to an active reward learning problem is similarly straightforward: we interpret R's communicative actions as questions and H's actions as answers. There is once again a simple generalization non-active reward learning.
Proposition 4. Every two- two-phase communicative assistance problem (M, T hooop' can be reduced an equivalent active reward learning problem.
Corollary 5. If a two-phase communicative assistance problem (M, ) has only one communica- tive robot action, it can be reduced an equivalent son-active reward learning problem.
4 QUALITATIVE IMPROVEMENTS FOR GENERAL ASSISTANCE
We have seen that reward learning is equivalent to two-phase communicative assistance problems, where inferring the reward distribution can be separated from control using the reward distribution. However, for general assistance games, it is necessary to merge estimation and control, leading to several new qualitative behaviors. When the two phase restriction is lifted, we observe relevance aware active learning and plans conditional on future feedback. When the communicative restriction lifted, we observe learning from physical actions.
We demonstrate these qualitative behaviors in simple environments using point-based value iteration (PBVI) or deep reinforcement learning (DRL). We describe the qualitative results here, deferring detailed explanations of environments and results to Appendix For communicative assistance problems, we also consider two baselines:
1. A Active reward learning. This is the reward learning paradigm discussed so far.
2. Interactive reward learning. This is a variant of reward learning that aims to recover some of the benefits of interactivity, by alternating reward learning and acting phases. During an action phase, R chooses actions that maximize expected reward under its current belief over 0 (without "knowing" that its belief may change), while during reward learning phase, R chooses questions that maximizes information gain.
4.1 PLANS CONDITIONAL ON FUTURE FEEDBACK
Here, we show how an assistive agent can make plans that depend obtaining information about 0 in the future. The agent can first take some "preparatory" actions that whose results can be used later once the agent has clarified details about A reward learning agent would not be able do this, as it would require three phases (acting, then learning, then acting again).
We illustrate this with our original kitchen environment (Figure D in which R must bake pie for H, but doesn't t know what type of pie H would like: Apple, Blueberry. or Cherry. Each type has weight specifying the reward for that pie. Assuming people tend to like apple pie the most and cherry pie the least, we have 0^ Uniform 2, 4), On Uniform[1, [1, 3). and Oc ~ Uniform(0, 2]. We define the questions Q = (qA, QB. } where qx means "What is the value of 0x?", and thus, the answer set is C R.
R can select ingredients to assemble the pic. Eventually, must use "bake", which bakes the selected ingredients into a finished pie, resulting in reward that depends what type of pie has been created. H initially starts outside the room, but will return at some prespecified time. ro assigns a cost of asking question of 0.1 if H is inside the room, and otherwise. The horizon is 6 timesteps.
Assistance. Notice that, regardless of H's preferences, R will need to use flour to make pie dough. So, R always makes the pie dough first, before querying H about her preferences. Whether R then

---

Under review as a conference paper at ICLR 2021
queries H about her preferences depends on how late H returns. If H arrives home before timestep 5. R will query her about her preferences and then make the appropriate pie as expected. However, H will arrive later, then there will not be enough time query her for her preferences and bake pie. Instead, R bakes an apple pie, since its prior suggests that that's 's what H wants.
This behavior, where R takes actions (making dough) that are robustly good but waits actions (adding the filling) whose reward will be clarified in the future, is very related to conservative agency (Turner et al. 2020), connection explored in more depth in Appendix D
Reward learning. The assistance solution requires R to act (to make dough), then to learn prefer- ences, and then to act again (to make pie). A reward learning agent only have two phases, and so we see one of two suboptimal behaviors. First, R. could stay in the learning phase until H returns home, then ask which pic she prefers, and then make the pic from scratch. Second, R could make an apple pie without asking H her preferences. (In this case there would be no learning phase.) Which of these happens depends on the particular method and hyperparameters used.
Interactive reward learning. Adding interactivity is not sufficient to get the correct behavior. Suppose we start with an action phase, The highest reward plan under R's current belief over to bake an apple pie, so that's what it will do, as long as the phase lasts long enough. Conversely, suppose we start with a learning phase. In this case, R does nothing until H returns, and then asks about her preferences. Once switch an action phase, it bakes the appropriate pic from scratch.
4.2 RELEVANCE AWARE ACTIVE LEARNING
Once we relax the two-phase restriction, R starts further optimize whether and when it asks questions. In particular, since R may be uncertain about whether a question's 's answer will even be necessary. R will only ask questions once they become immediately relevan! to the task at hand. In contrast, a reward learning agent would have to decide at the beginning of the episode (during the learning phase) whether or not to ask these questions, and so cannot evaluate how relevant they are.
Consider for example a modification to the kitchen environment: R knows that H wants apple pie, but when R picks up some apples, there is a 20% chance that it finds worms in some of the apples. R is unsure whether H wants her compost bin to have worms, and so does not know whether dispose of the bad apples in the trash or compost bin. Since this situation is relatively unlikely, ideally R would only clarify H's preferences when the situation arises.
The wormy-apples kitchen environ- ment. H wants an apple, but R might discover worms in the apple, and have dispose of in either of the trash or compost bins.
Figure 2:
Assistance. An assistive R only asks about wormy apples when it needs to dispose of one. R always starts by picking up apples. If the apple does not have worms, R immediately uses the apples bake the pie. If some apples have worms and the cost of asking a question is sufficiently low, R elicits H's preferences and disposes of the apples appropriately. then bakes the pie with the remaining apples.
This behavior, in which questions are asked only if they are useful for constraining future behavior, has been shown previously using probabilistic recipe trees (PRTs) Kamar et al. (2009), but to our knowledge has not been shown with optimization-based approaches.
Reward learning. reward learning policy must have only two phases and so would show one of two undesirable behaviors: either it would always ask H where to dispose of wormy apples, never asks and instead guesses when it does encounter wormy apples.
Interactive reward learning. This has the same problem as in the previous section. If we start in the action phase and R picks up wormy apples, will dispose of them in an arbitrary bin without asking H about her preferences, because it doesn't 't "know" that it will get the opportunity to do so. Alternatively, if we start with a learning phase, will ask H where to dispose of wormy apples, even if R would never pick up any wormy apples.
7

---

Under review as a conference paper at ICLR 2021
Note that more complex settings can have many more questions. Should R ask whether H would prefer to use seedless apples, should scientists ever invent them? Perhaps R should ask how her pie preferences vary based on her emotional state? Asking about all possible situations is not scalable.
4.3 LEARNING FROM PHYSICAL ACTIONS
So far we have considered commumicative assistance problems, in which H only provides feedback rather than acting maximize reward herself. Allowing H have physical actions enables a greater variety of potential behaviors. Most clearly, when R knows the reward (that is, Pe puts support over single 0), assistance games become equivalent to human-Al collaboration (Nikolaidis & Shah 2013 Carroll et al. 2019; Dimitrakakis et al. 2017).
With uncertain rewards, we can see further interesting qualitative behaviors: R can learn just by observing how H acts in an environment, and then work with H to maximize reward, all within single episode, as in shared autonomy with intent inference @Javdani et al. 2015; Brooks & Szafir 2019) and other works that interpret human actions as communicative Whitney et al. (2117). This can significantly reduce the burden on H in providing reward information to R (or equivalently, reduce the cost incurred by R in asking questions to H). Some work has shown that in such situations, humans tend to be pedagogic: they knowingly take individually suboptimal actions, in order to more effectively convey the goal to the agent (Ho al., [2116] Hadfield-Mence et al.) 2016). An assistive R who knows this can quickly learn what H wants, and help her accomplish her goals.
We illustrate this with a variant of our kitchen environment, shown in F There are no longer questions and answers. Both H and R can move to an adjacent free space, and pick up and place the various objects. Only R may bake the dessert. R uncertain whether H prefers cake or cherry pie.
For both recipes, it is individually more efficient for H to pick up the dough first. However, we assume H is pedagogic and wants to quickly show R which recipe she wants. So, if she wants cake, she will pick up the chocolate first to signal to that cake is the preferred dessert.
is not clear how exactly to think about this from a reward learning perspective: there aren't any communicative human actions since every action alters the state of the environment. In addition, there is no clear way separate out given trajectory into two phases. This situation cannot be easily coerced into the reward learning paradigm.
In contrast, an assistive R can handle this situation perfectly. It initially waits to see which ingredient H picks up first, and then quickly helps H by putting in the ingredients from its side of the environment and baking the dessert. learns implicitly to make the cake when H picks up chocolate, and to make the pie when H picks up dough. This is equivalent to pragmatic reasoning (Goodman & Frank] 2016): "H would have picked up the chocolate if she wanted cake, so the fact that she picked up the dough implies that she wants cherry pie". However, we emphasize that R is not explicitly programmed to reason in this manner, and is learned using deep reinforcement learning (Appendix C.3).
Figure 3: The cake-or-pic variant of the kitchen environment. H is equally likely to prefer cake or pie. Communication must take place through physical actions alone.
Note that R is not limited to learning from H's physical actions: R can also use its own physical actions to "query" the human for information (Woodward et al. 2019 Sadigh et al. 2016
5 LIMITATIONS AND FUTURE WORK
Computational complexity. The major limitation of assistance compared to reward learning is that assistance problems significantly more computationally complex, since we treat the unknown reward as the hidden state of a POMDP. We are hopeful that this can be solved through the application of deep reinforcement learning. An assistance problem is just like any other POMDP,
8

---

Under review as a conference paper at ICLR 2021
except that there is one additional unobserved state variable 0 and one additional observation a This should not be a huge burden, since deep reinforcement learning has been demonstrated to scale to huge observation and action spaces (OpenAI 2018 Vinyals et al., 2019). Another avenue for future work is to modify active reward learning a algorithms in order to gain the benefits outlined in Section 4 while maintaining their computational efficiency.
Increased chance of incorrect inferences. In practice, assistive agents will extract more information from H than reward learning agents, and it is worse if nh is misspecified. We don't see this as major limitation: to the extent this is a major worry, we can design n so that the robot only makes inferences about human behavior in specific situations. For example, by having z" be independent of 0 in given state s, we ensure that the robot does not make any inferences about 0 in that state.
Environment design. We have shown that by having a hidden human goal, we can design environ- ments in which optimal agent behavior is significantly more "helpful". One important direction for future work to design larger, more realistic environments, in order to spur research into how best to solve such environments. We would be particularly excited to see suite of assistance problems become a standard benchmark by which deep reinforcement learning algorithms are assessed.
5.1 LIMITATIONS OF ASSISTANCE AND REWARD LEARNING
While we believe that the assistance framework makes meaningful conceptual progress over reward learning, a number of challenges for reward learning remain unaddressed by assistance:
Human modeling. A major motivation for both paradigms is that reward specification is very difficult. However, now we need to specify a prior over reward functions, and the human model x Consequently, misspecification can still lead to bad results (Armstrong et al. [2020] Carey 2018). While it should certainly be easier to specify a prior over with a "grain of truth" on the true reward O" than to specify 0* directly, it is less clear that we can specify π well.
One possibility is to add uncertainty over the human policy π H However, this can only go so far: information about 0 must come from somewhere. If R is sufficiently uncertain about 0 and TH then it cannot learn about the reward (Armstrong & Mindermann 2018). Thus, for good performance we need to model n² While imitation learning can lead to good results (Carroll et al. 2019). the best results will likely require insights from a broad range of fields that study human behavior.
Assumption that H knows 0. Both assistance games and reward learning makes the assumption that H knows her reward exactly, but in practice, human preferences change over time (Allais 1979 Cyert & DeGroot 1975) Shogren al., (0000.. We could model this the human changing their subgoals (Michini & How [2012] Park et al. 2020)), adapting to the robot (Nikolaidis et al. 2017) or
learning from experience (Chan al. 2019).
Dependence on uncertainty. All of the behaviors of Section 4 as well as previously explored benefits such as off switch corrigibility (Haddield-Menelll et al. 2017a). depend on expecting to gain information about 0. However, It will eventually exhaust the available information about 0. If everything perfectly specified, this not a problem: R will have converged to the true 0*. However, in the case of misspecification, after convergence R is effectively certain in an incorrect 6 which has many troubling problems that we sought to avoid in the first place (Yudkowsky) yearr unknown).
6 CONCLUSION
While much recent work has focused on how we can build agents that learn what they should do from human feedback, there is not yet consensus on how such agents should be built. In this paper, we contrasted the paradigms of reward learning and assistance. We showed that reward learning problems are equivalent to a special type of assistance problem, in which the human may only provide feedback at the beginning of the episode, and the agent may only act in the environment after the human has finished providing feedback. By relaxing these restrictions, we enable the agent to reason about how its actions in the environment can influence the process by which it solicits and learns from human feedback. This allows the agent to (1) choose questions based on their relevance, (2) create plans whose success depends on future feedback, and (3) learn from physical human actions in addition to communicative feedhack.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Maurice Allais. The so-called allais paradox and rational decisions under uncertainty. In Expected utility hypotheses and the Allais paradox, pp. 437-681. Springer, 1979.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané, Concrete problems in Al safety. arXiv preprint arXiv:1606.06665 2016.
Brenna D Argall, Sonia Chermova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469 483, 2009.
Stuart Armstrong and Sören Mindermann. Occam's razor insufficient to infer the preferences of irrational agents. In Advances in Neural Information Processing Systems, pp. 5598-5609, 2018.
Stuart Armstrong, Jan Leike, Laurent Orseau, and Shane Legg. Pitfalls of learning a reward function online. arXiv preprint arXiv:2004. 13654, 2020.
Andrea Bajesy, Dylan Losey, Marcia K O' O'Mallyy, and Anca D Dragan. Learning robot objectives from physical human interaction. Proceedings of Machine Learning Research, 78:217-226, 2017.
Tirthankar Bandyopadhyay, Kok Sung Won, Emilio Frazzoli, David Hsu, Wee Sun Lee, and Daniela Rus. Intention- aware motion planning. In Algorithmic Foundations of Robotics pp. 475 491. Springer, 2013.
Erdem Bıyık, Malayandi Palan, Nicholas C Landolfi, Dylan P Losey, and Dorsa Sadigh. Asking easy questions: A user-friendly approach to active reward learning. arXiv preprint arXiv: 191.v.1910.04366 2019.
Nick Bostrom, Superintelligence: Paths, Dangers, Strategies. Oxford University Press, Inc., USA, 2014.
Connor Brooks and Daniel Szafir. Balanced information gathering and goal-oriented actions in shared autonomy. In 2019 14th ACM/IEEE International Conference on Stuman-Robre Interaction (HRI), pp. 20199,2011
Ryan Carey. Incorrigibility in the CIRL framework. In Proceedings of the 2018 AAAI/ACM Conference AI, Ethics, and Society, pp. 30-35, 2018.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in Neural Information Processing Systems, pp. 3174-5185 2019.
Lawrence Chan, Dylan Madfield-Menelle Siddhartha Srinivasa, and Anca Dragan. The assistive multi- armed bandit. In 2019 14th ACM/IEE International Conference on Tlmann-Rboot Interaction (HRI), pp. 354-363, IEEE, 2019.
Min Chen, Stefanos Nikolaidis, Harold Soh, David Hsu, and Siddhartha Srinivasa. Planning with trust for human-robot collaboration. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, pp. 307-315, 2018.
Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, 5(2):153-163, 2017.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pp. 4309_ 2017.
Jack Clark and Dario Amodei. Faulty reward functions the wild, 2016. URL httphhtpps//bllt/// openai com/ waaoo//aaltty-ewaaaaeee functions
Cohn, Robert W. Maximizing Expected Value of Information Decision Problems by Querying on a Wish-to-Know Basis. PhD thesis, University of Michigan, 2016.
10

---

Under review as a conference paper at ICLR 2021
Sam Corbett-Davies, Emmaa Pierson, Avi Feller, Sharad Goel, and Aziz Huq. Algorithmic decision making and the cost faimesss. In Proceedings of the 23rd SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 797-806, 2017.
Richard M Cyert and Morris H DeGroot. Adaptive utility. In Adaptive Economic Models, pp. 223-246. Elsevier, 1975.
Christian Daniel, Malte Viering, Jan Metz, Oliver Kroemer, and Jan Peters. Active reward learning. In Robotics: Science and systems, 2014.
Nishant Desai. Uncertain reward-transition mdps for negotiable reinforcement learning. 2017.
Christos Dimitrakakis, David C Parkes, Goran Radanovic, and Paul Tylkin. Multi-view decision processes: the helper-ai problem. In Advances in Neural Information Processing Systems, pp. 5443-5452, 2017.
Michael O Duff. Optimal learning: Computational procedures for Bayes-adaptive Markov decision processes. PhD thesis, University of Massachusetts Amherst, 2002.
Brochu Eric, Nando D Freitas, and Abhijeet Ghosh. Active preference learning with discrete choice data. In Advances in Neural Information Processing Systems, pp. 409-416, 2008.
Alan Fern, Sriraam Natarajan, Kshitij Judah, and Prasad Tadepalli. assistance. Journal of Artificial Intelligence Research, 50:71-104, 2014.
-teerrientthoortic model of
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce-
ment learning. arXiv preprint arXiv: 11248, 2017.
Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to goals: Inverse reinforcement learning for vision-based instruction following. arXiv preprint 1rXiv:1902.07742, 2019.
Yang Gao, Jan Peters, Antonios Tsourdos, Shao Zhifei, and Er Meng Joo. A survey of inverse reinforcement learning techniques. International Journal of Intelligent Computing and Cyhernetics, 2012.
Noah D Goodman and Michael C. Frank. Pragmatic language interpretation as probabilistic inference. Trends in Cognitive Sciences, 20(11 (0(11):818-82) 2016.
Dylan Hadfield-Menell, Stuart Russell, Pieter Abbeel, and Anca Dragan. Cooperative Inverse Reinforcement Learning. In Advances in Neural Information Processing Systems, pp. 3909-3917, 2016.
Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. The off-switch game, In Workshops at the Thirty-First AAAI Conference on Artificial Intelligence, 2017a.
Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design. In Advances Neural Information Processing Systems, pp. 6765-6774, 2017b.
Joseph Y Halpern and Rafael Pass. Game theory with translucent players. International Journal of
Game Theory, 47(3):949-976 2018.
Mark K Ho, Michael Littman, James MacGlashan, Fiery Cushman, and Joseph L Austerweil. Showing versus doing: Teaching by demonstration. In Advances Neural Information Processing Systems, pp. 3027-3035, 2016.
Shervin Javdani, Siddhartha S Srinivasa, and Andrew Bagnell. Shared autonomy via hindsight optimization. Robotics Science and Systems: online proceedings, 2015, 2015.
Hong Jun Jeon, Smitha Milli, and Anca Dragan. rationall (implicit) choice: A unifying formalism for reward learning. arXiv preprint arXiv:2002.04833 2020.
Ece Kamar, Ya' akov Gal, and Barbara J Grosz. Incorporating helpful behavior into collaborative plan- ning. In Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems (AAMAS). Springer Verlag, 2009.
11

---

Under review as a conference paper at ICLR 2021
Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair determi- nation of risk scores. arXiv preprint arXiv: (01.60000087,,7 2016.
Victoria Krakovna. Specification gaming examples in AI, 2018. URL htts::
wordpress http:///w////////////aaaaaaaaaa.
Joel Lehman, Jeff Clune, and Dusan Misevic. The surprising creativity of digital evolution. In Artificial Life Conference Proceedings, pp. 55-56. MIT Press, 2018.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: research direction. arXiv preprint arXiv: vrXiv:181.00777, 2018.
Owen Macindoe, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. POMCoP: Belief space planning for sidekicks cooperative games. In Eighth Artificial Intelligence and Interactive Digital Entertainment Conference, 2012.
Dhruv Malik, Malayandi Palaniappan, Jaime F Fisac, Dylan Hadfield-Menell Stuart Russell, and Anca D Dragan. An efficient, generalized Bellman update for cooperative inverse reinforcement learning, arXiv preprint 18Xiv: 03820, 2018.
James John Martin, Bayesian decision problems and Markov chains. Wiley, 1967.
Lucas Maystre and Matthias Grossglauser. Just sort it! A simple and effective approach to active preference learning. In Proceedings of the 34th International Conference Machine Learning, pp. 2024-234422 2017.
John McCarthy and Patrick J Hayes. Some philosophical problems from the standpoint of artificial intelligence. In Readings in Artificial Intelligence, pp. 431-450. Elsevier, 1981.
Bernard Michini and Jonathan P How. Bayesian nonparametric inverse reinforcement learning. In Joint European conference on machine learning and knowledge discovery in databases, pp. 148-163. Springer, 2012.
Sören Mindermann, Rohin Shah, Adam Gleave, and Dylan Hadfield-Menell. Active inverse reward design. arXiv preprint 1xXiv:8809003060 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013.
Andrew Ng and Stuart J Russell. Algorithms for inverse reinforcement learning. In International Conference on Machine learning, 2000.
Stefanos Nikolaidis and Julie Shah. Human-robot cross-training: computational formulation, mod- eling and evaluation of a human team training strategy. In 2013 8th ACM/IEEE International Conference Human-Robot Interaction (HRI), pp. 33-40. IEEE, 2013.
Stefanos Nikolaidis, Ramya Ramakrishnan, Keren Gu, and Julie Shah. Efficient model learning from joint-action demonstrations for human-robot collaborative tasks. In 2015 10th ACM/IEEE International Conference on Human- Staman-Robot Interaction (HRI), pp. 189-196. IEEE, 2015.
Stefanos Nikolaidis, David Hsu, and Siddhartha Srinivasa. Human-robot mutual adaptation in collaborative tasks: Models and experiments. The International Journal of Robotics Research, 36 (5-7):618-66) 2017.
Stephen M Omohundro. The basic AI drives. In Artificial General Intelligence, pp. 483-492, 2008.
OpenAL. OpenAl Five, 2018. htttee///ppeaat.ccc///ollgggggge
Dachyung Park, Michael Noseworthy, Rohan Paul, Subhro Roy, and Nicholas Roy. Inferring task goals and constraints using bayesian nonparametric inverse reinforcement learning. In Conference on Robot Learning, pp. 1004- 2020.
Joelle Pineau, Geoff Gordon, Sebastian Thrun, et al. Point-based value iteration: An anytime algorithm for POMDPs. In IJCAI, pp. 1025-1032, 2003.
12

---

Under review as a conference paper at ICLR 2021
Stephanie Rosenthal and Manuela Veloso. Modeling humans as observation providers using pomdps. In 2011 RO-MAN, pp. 53-58. IEEE, 2011.
Stuart Russell. Human Compatible: Artificial Intelligence and the Problem of Control. Penguin, 2019.
Dorsa Sadigh, Shankar Sastry, Sanjit A Seshia, and Anca Dragan. Information gathering actions over human internal state. 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 66-73. IEEE, 2016.
Dorsa Sadigh, Anca Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-based learning of reward functions. In Robotics: Science and Systems, 2017.
Rohin Shah, Dmitrii Krasheninnikov, Jordan Alexander, Pieter Abbeel, and Anca Dragan. Preferences implicit in the state the world. In International Conference on Learning Representations, 2019.
Jason F Shogren, John A List, and Dermot J Hayes. Preference learning in consecutive experimental auctions. American Journal of Agricultural Economics, 82(4):1016-1021 2000.
Alexander Matt Turner 1.Xi019912.0168,, 2019.
Optimal farsighted agents tend to seek power. arXiv preprint
Alexander Matt Tumer, Dylan Hadfield-Menel, and Prasad Tadepalli. Conservative agency via attainable utility preservation. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp. 385-391, 2020.
Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Woj- ciech M. Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, Timo Ewalds, Dan Horgan, Manuel Kroiss, Ivo Danihelka, John Agapiou, Junhyuk Oh, Valentin Dalibard, David Choi, Laurent Sifre, Yury Sulsky, Sasha Vezhnevets, James Molloy, Trevor Cai, David Budden. Tom Paine. Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Toby Pohlen, Yuhuai Wu, Dani Yogatama, Julia Cohen, Katrina McKinney, Oliver Smith, Tom Schaul, Tim- othy Lillierap, Chris Apps, Koray Kavukcuoglu, Demis Hassabis, and David Silver. Alphas- tar: Mastering the real-time strategy game StarCraft II. Iitps: /wwpmindddnnnee//ee/eeeedm/.mm com/blog//o/o////////o/////o/ alphastar mastering real - time strategy game - starcraftt 11/ 2019.
David Whitney, Eric Rosen, James MacGlashan, Lawson LS Wong, and Stefanie Tellex. Reduc- ing emrors object-fetching interactions through social feedback. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1006-1011 1013. IEEE, 2017.
Nils Wilde, Dana Kulic, and Stephen I. Smith. Active preference learning using maximum regret. arXiv preprint arXiv:2005.0400 044667 2020.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. survey of preference- based reinforcement learning methods. Journal of Machine Learning Research, 18(1):49454499, 2017.
Mark Woodward, Chelsea Finn, and Karol Hausman. Learning to interactively learn and assist, arXiv preprint 1711111906.10187, 2019.
Eliezer Yudkowsky. Problem of fully updated deference, year unknown. URL htttttt//aabbiaaa commp//pdated._cemm deference/
Shun Zhang, Edmund Durfee, and Satinder Singh. Approximately-optinal queries for planning in reward-uncertain markov decision processes. In l'went-Seventth International Conference Automated Planning and Scheduling, 2017.
Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. 2010.
13

---

Under review as a conference paper at ICLR 2021
A REWARD LEARNING AND ASSISTANCE FORMALISMS
A. RELATION BETWEEN NON-ACTIVE AND ACTIVE REWARD LEARNING
The key difference between non-active and active reward learning that in the latter R may ask H questions in order to get more targeted feedback. This matters as long as there is more than one question: with only one question, since there no choice for R to make, R cannot have any influence on the feedback that H provides. As result, non-active reward learning is equivalent to active reward learning with a single question.
Proposition Every non-active reward learning problem (Mrr, C, (O, re. Po), x k) can be reduced to an active reward learning problem.
Proof. We construct the active reward learning problem as (Myr Q', C, ra, Pe), n "1 R), where Q' A {go} where qs is some dummy question, and T (c q, a x (cl 0).
Suppose the solution to the new problem is (T J"). Since I' is a solution, we have:
f f argmax
ER( co.--)))
8~ S.PPo.ukk-nTt
Gook-
0))
FRR(f(JoX=1 = 20 co:k-1)) g are do
argmax
E
f 8~Pe 40ok Park rov (luo.0)
argmax
E
ER( = go: co:kk)))))
/ 8~ PoP (-10)
Thus ((co:-11) f f'000=1 = 4s. Co:k is maximizer of Egw Eo (-10) [R(f(cox-1) ) making it solution to our original problem.
Proposition 7. Every active reward learning problem LMrrQQ, Q,,, n" k) with [Q] can be reduced to non-active reward learning problem.
Proof. Let the sole question in Q be qg- We construct the non-active reward learning problem as (Mir, C, (0, ro, Pe), " k), with x HH' (c 0) = # qo. 0).
construct solution to the original Suppose the solution the problem is f" Then (0ss-1,0us-1) ) = 1 [[ qol. since there is R problem as follows. First, note that TQ must be π T8(4 only one possible question gş. Then by inverting the steps in the proof of Proposition 6
we can see
that f' is maximizer of Ea-Pe:90. 1-f8ira. (~
ER(f(: ca.--))) Thus, by defining
f 1, - (c) we get maximizer to our original problem, making (T f) solution to the original problem.
A.2 REDUCING ASSISTANCE PROBLEMS TO POMDPS
Suppose that we have an assistance problem (M, with:
M = (S, ( AR). (2 2 (O ORI.T,,..s.... (9, Pe)).
Then, we can derive single-player POMDP for the robot = (S' A" Ω' embedding the human reward parameter into the state. We must include the human's 's previous action "! into the state, so that the robot can observe it, and so that the reward can be computed.
O',, by
To allow for arbitrary Marn-aaroovian) human policies n could encode the full history in the state, in order to embed N into the transition function T. However, in our experiments we only consider human policies that are in fact Markovian. We make the same assumption here, giving policy " (af of & 0) that depends on the current observation and previous robot action.
The transformation M + M' is given follows:
14

---

Under review as a conference paper at ICLR 2021
S x
State space
0' 2 A
Observation space
O'(o' 8 = O'((o", , (s, 02 8))
Observation function
_ 1|a a 0"(0" s)
(' ay ((22, af 02) (s1, ap (2),a²)
Transition function
2 T(22 51, $1 at R 10² = 01l
Σ
O# (o 11)) - x o" ,)
aft EQH
( 8/2 = ((1) dn 0) a (32, 0))
Reward function
_ ro(s1, a a,a S2)
Ps((') = Pil((s a
,0))) .0) & Ps(s) (0) 1|a
Initial state distribution where a is arbitrary
In the case where the original assistance problem fully observable, the resulting POMDP is an instance of a Adyes-Adaptive MDP (Martin 1967 Duff 2002).
Any robot policy T can be translated from the APOMDP M naturally into an identical policy on M'. Note that in either case, policies are mappings from (Ω A# ARY x 12R to A(AR).
This transformation preserves optimal agent policies: Proposition 8. A policy R a solution of M if and only if it is solution of M'.
Proof. Recall that an optimal policy n" in the POMDP M' is one that maximizes the expected value:
EV(T)=
(r' (1), ar, (2+1) =
E
7'ro(s,, at ag s(+1)
where the trajectories r's sequences of state, action pairs drawn from the distribution induced by the policy, starting from state 8p.
Similarly, an optimal robot policy R in the APOMDP M is one that maximizes its expected reward:
ER ER( R)
E
Σ r'ooss af oB (641)
Ao Po 0.
To show that the optimal policies coincide, suffices to show that for any ग, ER(TT) (in M) is equal to FV(()) (in M'). To do this, we will show that induces the "same" distributions trajectories. For mathematical convenience, we will abuse notation and consider trajectories of the form 7; E (S, A A O; is easy translate trajectories of this form to trajectories in either M' or M.
the
We will show that the sequence 7; has the same probability when the robot takes the policy æ in both M' and M by induction on the lengths of the sequence.
First, consider the case of length sequences. :; [(s, a an ): 0). Under both M' and M, and O are drawn from Ps and Pe respectively. Similarly, al and a are drawn from π r²(- off) and H (- H R R²,0) respectively. So the distribution of length sequences is the same under both M and M.
Now, consider some longer sequence 7: [(s1, a ..... (St, a. a (] By the inductive hypothesis, the distribution of (#1, atf , _af))........)...... (8t at af aaf and 0 are identical; it suffices to show that (st, af at') has the same distribution, conditioned on the other parts of 7;0, under M' and under M. Yet by construction, s, is drawn from the same distribution T T, af=;), at drawn from the same distribution n²( of a. 8), and at is drawn from the same distribution R. 1o (R))))
15

---

Under review as a conference paper at ICLR 2021
A.3 OPTIMAL STRATEGY PAIRS POLICY-CONDITIONED BELIEF
We use the term conecy-conditioned belief to refer to a distribution over human policies which depends on the chosen robot policy. We use policy-conditineed beliefs as opposed a simple unconditional distribution over human policies, because it allows us to model wide range of situations, including situations with prior coordination, or where humans adapt to the robot's policy as a result of prior interactions. Moreover, this presents a unifying framework with prior work on assistance games (Hadfield-Mende) et al., 2016). In fact, finding an optimal strategy pair for the assistance game can be thought of as finding the policy which is best when the human adapts optimally, as formalized below:
Proposition Let M = AA AA (S 281 10", J.....ss (O. rg, Pa)) be an as- max EJR( x²)] be associated #"Ell"
sistance game. Let (TxR)(("))) 1[EJR(T" TR)
policy-conditoned belief. Let R be the solution to (M, B). Then (B(x BB) is optimal strategy pair.
Proof. Let (T " be arbitrary strategy pair. Then EJR(T" T < EJR(B(T " by the definition of B, and EJR(B(="), R) < (EJ((B(T)).. BB) by the definition of π². Thus EJR("" ) < EJR((((T))) T )) Since ( was assumed to be arbitrary, (B(T"), ), π is an optimal strategy pair.
B EQUIVALENCE OF RESTRICTED ASSISTANCE AND EXISTING ALGORITTMM
B. EQUIVALENCE OF TWO PHASE ASSISTANCE AND REWARD LEARNING
Here we prove the results Section 3 showing that two phase communicative assistance problems and reward learning problems equivalent.
We first prove Proposition and then use it to prove the others.
Proposition 4. Every two-phase communicative assistance problem (M, π ahoop) can be reduced equivalent active reward learning problem.
Proof. Let M = (S "A), ( 2 t0", DR,,T,Ps (s.,, (0. game, and let the assistance problem's 's action phase start at fact. Let a A be some arbitrary human action and of E = be some arbitrary human observation. We construct the new active reward learning problem Q', C², "0. Pa), """, R) as follows:
Pa)) be the assistance
a' fa E A : a R is communicative) C' A A"
Questions
Answers
M' & (S, R R² P(n))
POMDP
A' AR
Physical actions
8, a R (''s' 8, R) A T(s'
Transition function
2 tact
Number of questions
PT(s) A Σ PM SW+1 SV+1 =
H l o noop: a..k H
Initial state distribution
SovES
r06 R R,8') 2 ro(s, a R.
Reward function
-" " (c 0) a -" ="(cl 0 0)
"
Human decision function
Note that it is fine to use !f in T. ro and to use of in " even though they were chosen arbitrarily, because since the assistance problem is communicative, the result does not depend on the choice.
16

---

Under review as a conference paper at ICLR 2021
The PM term in the initial state distribution denotes the probability of trajectory under M and can be computed as
(M d0.2 a (s(
II T Se. az ,af).
Given some pair (=) assistance problem as
f²) to the active reward learning problem, we construct a policy for the
R (a. 1 abt--));
t << l and du E Q'
(af of o :
I' (a 10 (0 (a o a =
t and a dik-1 Q' and a E A'
else
We show that there must exist solution to P that is the analogous policy some pair. Assume towards contradiction that this is not the case, and that there is solution n R* that not the analogous policy some pair. Then we have few cases:
x It* assigns positive probability to at $ Q' for < k. This contradicts the two-phase assumption.
2. x assigns positive probability to af = q Q for i k. This contradicts the two-phase
assumption.
3. x Rt* (af -,,,T(1) depends on the value of of for some < Since both do:k-
HH
and R
00=-1 cannot affect the state or reward (as they are communicative), the distribution R over 0 is fixed and independent of π and so there must be some other π that is independent of o that does at least well. That x R would be the analogous policy to some pair, giving a contradiction.
Now, suppose we have some pair (T Rt' ff', and let its analogous policy be x ft Then we have:
E
[RR( (0
0~ 0~Paa.go 1
Cost M
E P~Pe
E
[ER(["(yox (1,
90ck Not
Posk "
EE
E
E
( nt s(+1)
8~Pe
98sk
Pook
1
E
E
E Su /o
1' of set1)
8~
98sk rost
Posk 1
(-1s,,af)
t=
1
E P~
E Quck-1 rost Cpsk- !f
Aroose ag az s041)
su ax 1. 1.0ma ""
t=k
However, since all the actions in the first phase are communicative and thus don't impact state or reward, the first & timesteps in the two phase assistance game have constant reward in expectation.
Let C
Eank k-1 "re( o aooop s141) This gives us:
E
[ER(J' ( (00 10))
D-Po- co 1
1
E
E Pm
y ro(s,, ae oa R
SE11)
P~Pa
1=0
(ER(TR)-
17

---

Under review as a conference paper at ICLR 2021
Thus, if (x0) f is solution to the active reward learning problem, then n R is solution of the two-phase communicative assistance problem.
Corollary 5. If a two-phase communicative assistance problem (M, H
" (hooo) ) has exactly one communicative robot action, it can be reduced to equivalent non-active reward learning problem.
(Note that the construction from Proposition does lead an active reward learning problem with a single question, meeting the precondition for Proposition 7
Proof. Apply Proposition A followed by Proposition 7
Proposition 2. Every active reward learning problem P = (M,Q,C, ra, Pe), n" k) can be reduced to equivalent two phase communicative assistance problem P' = (M', n ""))
Q be some question and co E C be some (unrelated) choice. Let N be a set of fresh states (no. number of questions asked so far. Then, we construct the new two phase communicative assistance problem p = (M', T aRoop) follows:
Proof. Let M = (S, 7). Let "Ro.....4-11) -111):
we will use these to count the
M' & (S', (C,AR) 1." .2 3, ,0 OF
Assistance game
S" SUN
State space
P8(()) _ 1 = no]
Initial state distribution
A & AUQ
Robot actions
2 S
H''s observation space
9 & nUN
R's observation space
8) A 1 8]
Hi's observation function
[1 = 8), SEN
OR RR
8) A
H's observation function
1OOOR 18, else
Po(()),
8 Rg-
1[8' n m441],
$ n, with i k
('(5') 1 R
Transition function
T(8' 8, a )
8 E S and a R E A,
1[s' = s),
else
& E N and a & Q,
00,
$ E S and a E Q. E N and fl E Q,
H ,a 8') A
00,
Reward function
0,
ro(s, a R'),))
else
n (a | 0), a EQ
H1'
H (a
A
Human policy
Co:
else
nooop
go
Distinguished noop action
Technically r' should not be allowed to return -o0. However, since S and A are finite, rp is bounded, and so there exists some large finite negative number that is functionally equivalent to oo that we could use instead.
see T and r' are independent of a and n. II' is independent of " making this a communicative assistance problem. By inspection, we can see that every q E Q is communicative robot action. Any a & Q must not be communicative action, because the reward rg differs between a R and qu- Thus, the communicative robot actions are Q and the physical robot actions are A.
Looking the definitions,
Note that by construction of P and T. we must have s; = n; for E 10...... k 1}. after which 8g is sampled from Po and all E S for > k. Given this, by inspecting r/y we can that
18

---

Under review as a conference paper at ICLR 2021
optimal policy must have d0:k-1 Q and a. & Q avoid the -o rewards. Since a & Q, have d co- Thus, setting @noop co. have that the assistance problem is two phase with actions at tact = k, as required.
Let policy "² for the assistance problem be reasonable if it never assigns positive probability to a E A when < k or to a" E Q when & 2 k. Then, for any reasonable policy rt we can construct an analogous pair ( f) to the original problem P follows:
T Qu:i-1: Cosi -
R (q: 00:-1 no:-1, 00-1 90:00 00:-- Co:i-1);
1 co:k (ax oo:r; G0:--1) 1) - (as
0ultk
R
a R
10:4 -. 10:+k-1)
where for the second equation we have
f
R
H ao:k
Ozk
71ock11
do 90:k-1
Co:k
R a k ag:t-1
Og:t
kkt+k-1 noop
Note that this is a bijective mapping.
Consider some such policy x and its analogous pair (TQ. f)- By construction of T, we have that the first k states in any trajectory are Ro:k-1 and the next state is distributed as Po(-). By our assumption on n we know that the first k robot actions must be selected from Q and the remaining robot actions must be selected from A, which also implies (based on π =²) that after the the remaining human actions must be co. Finally, looking at rg Thus:
that the first & timesteps get reward.
ERp (7: RD
rroolh of R sataa
E 0 3n)
Σ
r'ro(si, aff af s6-1)
9mPh Pm.aaa N ,a wot of - (o0, Rng
Li=k
rroos ae, 821)
E Po 00st wat- you Pa. Po =g
1=0
0~Pe 0-
which is the objective of the reward learning problem scaled by *k
Since we have a bijection between reasonable policies in P' and tuples in P that preserves the objectives (up constant), given solution -. fi+ P' (which must be reasonable), its analogous pair (m f') must be solution to P.
Corollary 3. Every non-active reward learning problem (M, C, (0, re. Pe), T k) can be reduced to an equivalent two phase communicative assistance problem (M', H
Proof. Apply Proposition o
followed by Proposition
B.2 ASSISTANCE WITH NO REWARD INFORMATION
In a communicative assistance problem, once there is no information to be gained about 0, the best thing for R to do is to simply maximize expected reward according to its prior. We show this in the particular case where x" is independent of 8 and thus cannot communicate any information about #:
Proposition 10. communicative assistance problem (M, π n²) where π " independent of 0 can be reduced POMDP M' with the same state space.
Proof. Given M (S, ( AR)
1 O" D,I.T,P. (0, ra, Pe)) we define a new POMDP as M (5, Ω O T', Ps, 1) with (s' s.a T(s' s, a a and
(1( R ,,) = Ea-p, ro((, H R 8')
Here, a is some action in A note that it does not
19

---

Under review as a conference paper at ICLR 2021
matter which action is chosen since in communicative assistance problem human actions have impact on T and r.
Expanding the definition of expected reward for the assistance problem, get:
ER(
E Pa & Pe i
Σ rrooss, at of 80+1)
0,
E 80
Σ rroos af of s441)
8~PP
Tm (50,0,
Note that because n. (a H1 ot a ( 0) is independent of 0, the robot gains no information about and thus x is also independent of 0. This means that have:
ER(T
E 80
Σ rroos,, at af st+1)
Pr
(50.
Let F T'man max s a o' no (s a" aa R s')l
(which exists since S, A" and A are finite). Then:
Mrolsu 8')1 I'Tmaz
mmaz
So apply Fubini's theorem to swap the expectations and sums. Applying Fubini's theorem twice gives us:
ER n)
= E
E Fro
E B
r'ross ! af st41)
80 Pp
Rt)
t=0
E
E Tr
E 0~-
[ralse, at at st+1)]
R)
E Pp
E Fro
(st, Rt s241)
In addition, the trajectories are independent = since the assistance problem is communicative, and so for a given policy т², the trajectory distributions for M and M' coincide, and thus the expected rewards for N also coincide. Thus, the optimal policies must coincide.
C EXPERIMENTAL DETAILS
C.I PLANS CONDITIONAL ON FUTURE FEEDBACK
In the environment described in Section 4.1 It needs to bake either apple or blueberry pie (cherry is never preferred apple) within 6 timesteps, and may query H about her preferences about the pie. Making the pic takes 3 timesteps: first R must make flour into dough, then it must add one of the fillings, and finally it must bake the pic. Baking the correct pie results wrong one results in penalty of -1. In addition, H might be away for several timesteps at the start of the episode. Querying H costs 0.1 when she present and when she is away.
reward, while baking the
The optimal policy for this environment depends on whether H would be home early enough for R to query her and bake the desired the pie by the end of the episode. R should always quickly make dough, as that is always required. If H returns home on timestep or earlier, R should wait for her to get home, ask her about her preferences and then finish the desired pie. If returns home later, should make its best guess about what she wants, and ensure that there is pic ready for her to cat: querying H when she is away is too costly, and there is not enough time to wait for H, query her, put in the right filling, and bake the pie.
We use PBVI to train an agent for this assistance problem with different settings for how long H is initially away. As expected, this results in policy that makes dough, queries H and bakes the correct pie if is back on timestep or earlier; if is back on timestep or 6, R simply makes apple pie that is most likely to be what H wants.
20

---

Under review as a conference paper at ICLR 2021
CakeOrPie Environment
pedagagic optimal R
Average Return
non-pedagogia optimal R
2
4
5
Timesteps
Figure 4: DQN smoothed learning curves on the CakeOrPic environment, with 6 seeds over 5.M
timesteps and learning rate of 10-
C.2 RELEVANCF-AWARE ACTIVE LEARNING: OPTIMAT. GRIDWORLDS
In the wormy-apple environment described in Section 4.2 the robot had to bring the human some apples in order to make a pie, but there's 's a 20% chance that the apples have worms in them, and the robot does not yet know how to dispose of soiled apples. The robot gets 2 reward for making an apple pie (regardless of how it disposed of any wormy apples), and gets - reward if it disposes of the apples in the wrong container. Additionally, asking a question incurs cost of 0.1. We solve this environment with exact value iteration.
If the environment two-phase, with a lower discount rate = 0.9), R's policy never asks questions and instead simply tries to make the apple pie, guessing which bin to dispose of wormy apples in it encounters any. Intuitively, since it would have to always ask the question at the beginning, would always incur cost of 0.1 as well as delay the pie by timestep resulting in 10% less value, and this is only valuable when there turn out be worms and its guess about which bin dispose of them in is incorrect, which only happens 10% of the time. This ultimately isn't worthwhile. This achieves an expected undiscounted reward of 1.8. Removing the two-phase restriction causes R to with this low discount. With this result achieves the maximal
ask questions mid-trajees mid-trajectory, expected undiscounted reward 1.98.
With higher discount rate of À 0.99, the two-phase policy will always ask about which bin to dispose of wormy apples in, achieving 1.9 expected undiscounted reward. This is still less than the policy without the two-phase restriction, which continues get undiscounted reward 1.98 because it avoids asking question 80% of the time, and so incurs the cost of asking a question less often.
C.3 LEARNING FROM PHYSICAL ACTIONS: CAKE-OR-PLE EXPERIMENT
In the environment described in Section 4.3 H wants dessert, but R is unsure whether H prefers cake or pic. Preparing the more desired recipe provides base value of V = 10, and the less desired recipe provides a base value of V 1. Since H doesn't 't want the preparation take too long, the actual reward when dessert made is given by rr V f(t), with f(t) as the episode horizon.
(1/N)*, and = 20
The experiments use the pedagogic H, that picks the chocolate first if they want cake, which allows R. to distinguish the desired recipe early this is in contrast with the non-pedagogic H, which does not account for R beliefs and always goes for the dough first.
With the pedagogic H. the optimal It does not move until Hl picks or skips the dough; if Hl skips the dough, this implies the recipe is cake and R takes the sugar, and then the cherries otherwise it goes directly for the cherries. With the non-pedagogic H. the optimal R goes for the cherries first (since it a common ingredient), and only then it checks whether H went for the chocolate or not, and has to go all the way back grab the sugar if H got the chocolate.
21

---

Under review as a conference paper at ICLR 2021
We train R with Deep Q-Networks (DQN; (Mnih et al. 2013 ) ran seeds for 5M timesteps and a learning rate of 10 results are shown in Figure
D OPTION VALUE PRESERVATION
In Section 4.1, we showed that R takes actions that are robustly good given its uncertainty over 0, but waits on actions whose reward will be clarified by future information about O. Effectively, R is preserving its option value: it ensures that it remains capable of achieving any of the plausible reward functions it is uncertain over.
related notion is that of conservative agency (Turner et al. 2020), which itself aims to preserve an agent's s ability to optimize wide variety of reward functions. This is achieved via attainable utility preservation (AUP). Given an agent optimizing a reward rsppe and a distribution over auxiliary reward functions Y Taux. the AUP agent instead optimizes the reward
T TA a) = T'spec(s: a) - X E [max (s, (g) - Qr... (8, a),0))
Yau
where the hyperparameter À and ag is an action that corresponds to R "doing nothing".
determines how much to penalize action for destroying option value,
However, the existing AUP penalty is applied to the reward, which means it penalizes any action that is part of a long-tern term plan that destroys option value, even the action itself does not destroy option value. For example, in the original Kitchen environment of Figure [iith sufficiently high X, any trajectory that ends with baking a pie destroys option value and so would have negative reward. As result, there is no incentive make dough: the only reason make dough to eventually make pie, but we have established that the value of making a pie is negative.
What we need is to only penalize action when it is going immediately destroy option value. This can be done by applying the penalty during action selection, rather than directly to the reward:
π AU (s) argmax Qrepe (s, a) X E Imax(Qraa (s, a.) Qraa (s, a), 0)]
Pas
After this modification, the agent will correctly make dough, and stop since it does not know what filling to use.
In an assistance problem. R will only preserve option value if expects to get information that will resolve its uncertainty later: otherwise, it might as well get what reward it can given its uncertainty. Thus, we might expect to recover existing notions of option value preservation in the case where the agent initially uncertain over 0, but will soon learn the true 0. Concretely, let us consider fully observable communicative Assistance POMDP where the human will reveal @ on their next action. In that case, It's chosen action a gets immediate reward f(s, a) Es [ro(s, a)], and future reward Pa T({ssa)) (s'),, where Vo refers to the value of the optimal policy when the reward is known to be rg and the initial state is s. Thus, the agent should choose actions according to:
E (rro(s,a) (1o((sa)++7V6(6))) +V66661
argmax
E of (e
F(s,a) (ls,a)+(Vis +VV(''') - -V-' [Vo
argmax
E T((saa)
argmax Q+(3, TE
E [V<(8')]
E d'~T(-la.a)
[Vo('']
This bears many resemblances to the AUP policy, once we set the distribution over auxiliary rewards to be the distribution over along with Tspec = f and X ==y, y. Nonetheless, there are significant differences, primarily because AUP was designed for the case where rspee and Taux could be arbitrarily different, which is not the case for us. In particular, with AUP the agent is penalized for any loss in Taux by taking the chosen action a relative to doing nothing, while in the assistance problem, the agent penalized for any loss in rgg by acting according to relative to what could be achieved if R knew the true reward. It is intriguing that both these methods lead to behavior that we would characterize "preserving option value".
22


Reviews:
[{"review_id": "xABc0UhVmV", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis work proposes learning a single control policy for human-in-the-loop learning rather than having a reward learning component and a control component. The key difference is that the action selection can use information from the reward learning module. The authors formulate an assistance game in this setting and show that it can reduce to an equivalent POMDP. The work then describes a communicative assistance problem and shows the equivalence of reward learning to assistance and visa versa. Results show qualitative improvements on variants of the kitchen domain.\n\nPros:\n- The paper was overall well-written (although some key differences with prior work were unclear, as described below).\n- The high-level area of human-in-the-learning is a useful and important space.\n\nCons:\n- There are many works in human-in-the-loop learning, and it wasn’t clear whether the ideas in this paper were novel enough at a high-level. Some relevant works are included below.\n- The paper was hard to follow with respect to the distinction between reward learning and assistance. Adding more examples or describing the difference from other angles could be useful.\n- The experimental domains were simple and there were no computational results shown in the main paper. This is important to show in order to support the claims of the paper.\n\nComments:\n- The qualitative results are useful for showing specific cases in which the proposed approach may be beneficial but this restricts how applicable the approach seems to be. It would be nice to see computational results over a variety of domains and comparisons with baseline approaches to see the benefit of the approach more generally. \n- Section 4.2 focuses on asking the right questions at the right time. What is the main novelty here with respect to more general active learning approaches?\n- A few minor notational confusions: for example, it’s confusing to have R be the robot as it’s often the reward function. And in Section 4.1, C is used for choices and for cherry.\n- Other work that might be relevant to the problem proposed here:\n\nGame-theoretic modeling of human adaptation in human-robot collaboration\nS Nikolaidis, S Nath, AD Procaccia, S Srinivasa\n\nMaximizing BCI Human Feedback using Active Learning\nZ Wang, J Shi, I Akinola, P Allen\n\nModeling humans as observation providers using pomdps\nS Rosenthal, M Veloso\n\nActive Learning for Risk-Sensitive Inverse Reinforcement Learning\nR Chen, W Wang, Z Zhao, D Zhao\n\nContact: Deciding to communicate during time-critical collaborative tasks in unknown, deterministic domains\nVV Unhelkar, JA Shah\n\nEfficient model learning from joint-action demonstrations for human-robot collaborative tasks\nS Nikolaidis, R Ramakrishnan, K Gu, J Shah\n\nRecommendation:\nOverall, I thought the work at a high-level was limited in its novelty compared with prior work in human-in-the-loop learning. The distinction between reward learning and assistance, a key part of the paper, was hard to fully understand so clarifying this description through examples or more clear text would be valuable. The evaluation was also simple and not very convincing with respect to supporting the claims. Adding computational experiments and appropriate baselines would be important to show the general applicability of the approach.\n\n--------------------\nResponse after rebuttal:\nThank you to the authors for their response. I appreciate the detailed answers to each of the prior works. I still do think the paper needs to be more clear in the problem and differentiation with prior work in the writing itself, which will require a non-trivial update to the paper.\n\nOn the computational results + baselines side, while the authors have run the experiment and have described these qualitative behaviors, this isn't a substitute for quantitative results, especially because this is important to show when comparing with baselines. The authors say \"We have updated Section 4 to be clearer about what baseline approaches would do in the environments we have tested\". It's important to show evidence that this is what the baselines actually did.\n\nBased on these points, I don't think the paper is quite ready for publication yet.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IsJJ_J7rcBM", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Overview\n\nThe paper addresses the problem of learning from human feedback. It provides an analysis of reward learning---where human feedback is used to extract a task description in the form of a reward---and assistance---where the learning agent and human co-exist in the environment and both perform actions; the agent seeks to select its actions to optimize the (unknown) that is implicit in the human's actions. \n\nThe paper shows that reward learning problems can be converted to assistance problems, turning queries from reward learning to communicative actions in a two-phase communicative assistance problem. Conversely, two-phase communicative assistance problems can be converted to active reward learning problems.\n\n### Comments\n\nIn my opinion, the paper reads very well, and the discussion in the paper is quite interesting. \n\nIn spite of an interesting discussion, I am not certain about the contribution of the paper. The derivations, although they seem technically sound, are not particularly surprising; similarly, the qualitative differences in behavior pointed out in the paper are hardly surprising---assistance learning, being a more general problem than reward learning (as follows from the discussion of the paper), will lead to more diverse behaviors. \n\nThe particular behaviors observed are also a consequence of the POMDP formulation, as the agent will act to strike an adequate tradeoff between information gathering and goal optimization.\n\n### Post-rebuttal update\n\nI thank the authors for the clarifications and discussion. However, and admitting that I may have missed something, I remain unconvinced regarding the contributions of the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DcvDLwyij8r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\n\nThe submission provides a survey of two paradigms for ‘agents learning from human feedback.’ The two paradigms are unified under a new formalism (assistance games), which subsumes them as its special cases. Further, a taxonomy of different problems resulting from the formalism is provided (communicative games, two-phase games, etc.), along with illustrative examples of resulting agent behaviors. Based on the survey and taxonomy, the authors highlight that the assistance paradigm is more advantageous (in terms of possible behaviors that it can result in) than the reward learning paradigm.\n\n=======================================================\n#### Reasons for score\n\nStrengths\n  + The topic of ‘agent learning from human feedback’ is topical and of interest to the ICLR community.\n  + The proposed taxonomy (i.e., assistance games) can serve as a useful common ground for discussing the different paradigms, problems, and solutions of ‘agent learning from human feedback.’\n  + The paper is well written and organized.\n\nWeaknesses\n- While the submission does discuss related work (Section 2), the discussion omits several related research threads. Some of these threads have strong overlap with the setting proposed in the submission.\n- Similarly, the qualitative behaviors that emerge from assistance games (Section 4) have been demonstrated in prior research (including on larger problems and with human users), thereby making it difficult to assess the novelty of the formalism.\n- The key contribution of the submission is unclear (e.g., whether it is a survey, a model, a taxonomy, or all?).\n \nI am truly on the fence regarding this submission. A novel taxonomy is certainly needed to relate and compare the diverse and growing body of research in the area of ‘agent learning from human feedback.’ However, to arrive at this taxonomy a more complete consideration of existing formalisms and algorithms is necessary. Please see suggestions listed below on prior research that relates to and, in certain cases, extends the formalism of assistance / collaboration.\n\n=======================================================\n#### Key Comments\n\n1.\t(Introduction and Proposition 1) The insight of having a single control policy for both reward learning and control modules has been previously explored. This insight is identical to that of planning / control formulations in human-robot interaction literature that model the human’s preferences (which in turn influence the reward) as latent states. These planning methods use POMDPs to represent the interaction / collaboration problem, and solve the exploration-exploitation tradeoff associated with reward learning (exploration) and control (exploitation) using POMDP solvers or MPC. Please discuss the novelty of the proposed formalism in relation to these methods. For instance, \n    - (considers continuous states and actions spaces; assistance paradigm) Sadigh, Dorsa, et al. \"Information gathering actions over human internal state.\" 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2016.\n    - (does not require one agent, R or H, to act before the other; assistance paradigm) Chen, Min, et al. \"Planning with trust for human-robot collaboration.\" Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction. 2018.\n    - Gopalan, Nakul, and Stefanie Tellex. \"Modeling and solving human-robot collaborative tasks using pomdps.\" RSS Workshop on Model Learning for Human-Robot Communication. 2015.\n    - Nikolaidis, Stefanos, et al. \"Game-theoretic modeling of human adaptation in human-robot collaboration.\" Proceedings of the 2017 ACM/IEEE international conference on human-robot interaction. 2017.\n\n2. (Section 2.3) Assistance games, as defined, assume parametric specification of the hypothesis space of the reward / preference of humans. However, nonparametric extensions (both for assistance and reward learning) have been proposed. Please consider relating the proposed formalism with these prior works. For instance, \n  - Michini, Bernard, and Jonathan P. How. \"Bayesian nonparametric inverse reinforcement learning.\" Joint European conference on machine learning and knowledge discovery in databases. Springer, Berlin, Heidelberg, 2012.\n  - Panella, Alessandro, and Piotr Gmytrasiewicz. \"Bayesian learning of other agents' finite controllers for interactive POMDPs.\" Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. 2016.\n\n3. (Section 2.3) In assistance games, as defined, the reward (or human preferences over reward) does not change during the task. However, extensions exist which model the latent state corresponding to reward as being locally active and / or time varying. Please consider relating the proposed formalism with these related works. For example, \n  - (locally active reward; reward learning paradigm) Michini, Bernard, and Jonathan P. How. \"Bayesian nonparametric inverse reinforcement learning.\" Joint European conference on machine learning and knowledge discovery in databases. Springer, Berlin, Heidelberg, 2012.\n  - (locally active reward; reward learning paradigm) Park, Daehyung, et al. \"Inferring Task Goals and Constraints using Bayesian Nonparametric Inverse Reinforcement Learning.\" Conference on Robot Learning. PMLR, 2020.\n  - (considers time varying preferences with learned dynamics; assistance paradigm) Nikolaidis, Stefanos, David Hsu, and Siddhartha Srinivasa. \"Human-robot mutual adaptation in collaborative tasks: Models and experiments.\" The International Journal of Robotics Research 36.5-7 (2017): 618-634.\n  - (considers time varying preferences with learned dynamics; assistance paradigm) Unhelkar, Vaibhav V., Shen Li, and Julie A. Shah. \"Semi-Supervised Learning of Decision-Making Models for Human-Robot Collaboration.\" Conference on Robot Learning. 2020.\n\n4.  The behaviors arising from solving assistance games (Section 4) have been previously formalized by multiple human-AI collaboration approaches and demonstrated with human users. For instance,\n  -\t(exhibits behaviors outlined in Section 4.2) Kamar, Ece, Ya’akov Gal, and Barbara J. Grosz. \"Incorporating helpful behavior into collaborative planning.\" Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems (AAMAS). Springer Verlag, 2009.\n  -\t(reasons about communicative actions) Whitney, David, et al. \"Reducing errors in object-fetching interactions through social feedback.\" 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2017.\n  -\t(reasons about both physical actions and communications) Nikolaidis, Stefanos, et al. \"Planning with verbal communication for human-robot collaboration.\" ACM Transactions on Human-Robot Interaction (THRI) 7.3 (2018): 1-21.\n  -\t(reasons about both physical actions and communications) Unhelkar, Vaibhav V., Shen Li, and Julie A. Shah. \"Decision-Making for Bidirectional Communication in Sequential Human-Robot Collaborative Tasks.\" Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction. 2020.\n  -\t(communicative actions) Liang, Claire, et al. \"Implicit communication of actionable information in human-ai teams.\" Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 2019.\n\nPlease discuss the connection of the proposed formalism with these approaches.\n\n5.\t(Section 2.2) Please also consider discussing the following related works on active reward learning,\n  - Lopes, Manuel, Francisco Melo, and Luis Montesano. \"Active learning for reward estimation in inverse reinforcement learning.\" Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, Berlin, Heidelberg, 2009.\n  - Brown, Daniel S., Yuchen Cui, and Scott Niekum. \"Risk-aware active inverse reinforcement learning.\" Conference on Robot Learning. 2018.\n  - Tschiatschek, Sebastian, et al. \"Learner-aware teaching: Inverse reinforcement learning with preferences and constraints.\" Advances in Neural Information Processing Systems. 2019.\n  - Cui, Yuchen, and Scott Niekum. \"Active reward learning from critiques.\" 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018.\n\n=======================================================\n#### Minor Comment\n\n- (Section 2.1) Please consider including a note explaining the asterisk notation (used to define the domain for POMDP policy).\n\n=======================================================", "extra_scores": {}, "extra_text": {}}, {"review_id": "BF2Wh9JwPo8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "UPDATE:\nAfter extensive discussion with the authors, I'm raising my score to a 7.  \nI believe their revision will adequately address the concerns I've raised.\nI think this paper clearly identifies and illustrates the qualitative advantages of assistance, and that this is a novel and significant contribution.\n\nIn particular, I do *not* believe, as other reviewers seem to, that any of the following are sufficient reasons for rejecting this work:\n* The wealth of prior work on variants of reward learning and assistance \n* The lack of a comprehensive survey or categorization of such work in this submission\n* The lack of further results\n\nAfter reading the other reviews and responses, I am more confident that this paper makes a valuable contribution, although I stand ready to be challenged by other reviewers.  This is because the authors have argued that the qualitative benefits they describe in sections 4.1/2/3 have not been available to any of the many previous works reviewers mentioned, and no reviewers disputed this.  Furthermore, I did not find the reasons provided for rejection to be very relevant to the goals of this work.  So overall, I do not believe that other reviewers have made a strong case for rejecting this work.\n\nIn my mind, the best argument would seem to be simply that the contribution is insufficient.  I think this is a common criticism of papers that do not adhere to a conventional format or \"type\", but in this case, it seems unfair.  I believe the intellectual contribution of this paper is rather modest, but nonetheless novel and significant.  And I found this motivation for the work quite compelling (emphasis mine):\n> This existing literature is exactly why we wrote this paper: almost all of the existing literature on learning without reward functions can be captured by the reward learning paradigm as we formalize it. But (as we show) the assistance paradigm can enable significantly better behavior from the agents we train! **We are hoping to influence researchers to put more effort into algorithms for the assistance domain, in order to realize these qualitative benefits, instead of continuing to work in the reward learning paradigm as they have done so far.**\n\nI would encourage the authors to explain this goal in their revision, and make sure their claims about the superiority of assistance are appropriately modest.  Overall, I think the qualitative benefits of assistance presented provide a compelling argument for more work in the assistance paradigm, *given the paucity of such work*.  But I think the overall message of the paper should be: \"Given these advantages, and the lack of work on assistance, there should be more work on assistance, since it seems promising and neglected\", and not \"Assistance is better, so why would you do reward learning?\"  And my first impression of the paper was closer to the latter.  \n\nEND UPDATE\n\n-------------------------\nEvaluation:\n\nThis paper is well written and makes a nice point regarding qualitative advantages of assistance over reward learning.  Specifically, the authors show how assistance naturally leads an agent to \"agent to (1) choose questions based on their relevance, (2) create plans whose success depends on future feedback, and (3) learn from physical human actions in addition to communicative feedback.\"  However, the framing is quite one-sided, and the authors make no mention of potential advantages of reward learning.  And it is possible to achieve the same qualitative advantages by slightly modifying the (rather restrictive) formulation of reward learning that the authors use.  I think the work should either include such a discussion, or offer more convincing evidence regarding when assistance is in fact preferable in practice (e.g. experiments with some features that might advantage either approach).\n\nSome potential advantages of reward learning are:\n- reduced complexity of the learning problem\n- the human retains more control\n- it seems to require less modeling of human psychology\n- less opportunities to corrupt the reward signal\n\nRegarding the qualitative advantages mentioned in Sections 4.1/2, we can achieve the same benefits in many instances by incorporating regular feedback sessions where H and R can communicate.  I was confused by the claimed advantage in 4.3, since IRL already can learn from physical actions.\n\nOverall, I think this paper sets up a bit of a false dichotomy between (non-interactive) reward learning, and the full assistance game formulation.  We can instead view these methods as varying wrt 1) when and how we ask R to interpret H's actions as communicative, and 2) how interactive the learning process is.\nThis paper still makes important (corresponding) points: 1) there is a benefit to considering H's behavior as communicative whenever we understand its semantics, and 2) interaction is important.\n  \nThese are not very surprising, although very nicely and clearly argued for and demonstrated. However, regarding point (1): if we don't understand the semantics of H's behavior, it makes sense to restrict the communication to a more well-defined channel, as in reward learning; furthermore, even if we *do* know the semantics, H may not wish all of her behavior to be viewed as communicative, and allowing her to directly control when and how her behavior will be viewed as communicative grants H more agency over the behavior of R.\n\nAccepting points (1) and (2), it still seems like most effective methods are likely to lie somewhere between the two extremes described in this work.  For example, the authors state: \"Thus, for good performance we need to model π^H. This will require insights from a broad range of fields that study human behavior.\"  However, accurately modelling π^H  seems significantly less urgent when communication from H is more limited and literal (i.e. when we move closer towards current reward learning practice).  As another piece of loosely supporting evidence, in \"Literal or Pedagogic Human? Analyzing Human Model Misspecification in Objective Learning\", Milli and Dragan show that more sophisticated assumptions about the semantics of human behaviors may also be more brittle.\n\nAnother contribution of the paper is to explicitly frame reward learning as a special case of assistance.  This seems like a straightforward, minor, contribution.\n\nI was also disappointed that the paper didn't discuss reward corruption. The need to more fully understand H in the assistance paradigm creates more opportunities for R to misinterpret H's behavior.  In practice, the tighter feedback loops favored by the assistance paradigm might also create more opportunities for R to (e.g. irreversibly) \"corrupt\" H's reward function and/or policy, since an initial misunderstanding could be self-reinforcing.  For instance, H might fail to tell R that its behavior was intimidating if doing so previously had led R to become more (instead of less) intimidating.  This could lead R to become confident that this behavior was *not* intimidating to H.  Such a scenario seems less likely when H provides feedback to R in an \"offline\" or \"purely communicative\" context.\n\n-------------------------\nDetailed Suggestions:\n- Figure 1 caption could be clearer (what is depicted on the right?)\n- \"These behaviors cannot be expressed by a reward learning agent.\" <-- not literally true; suggest a rephrase.\n- \"Since c0:k-1 only serves to provide information to R\" <-- It doesn't have to... e.g. people do IRL on data that doesn't fit this description.\n- Provide more of an informal introduction in section 2.3 before jumping into definitions\n- \"However, since we want to compare to reward learning, we follow its assumptions and so assume that the human policy πH is available.\" While I was eventually able to understand this reasoning, it sounds false out of context, and should be clarified.\n- a^H_noop is not defined\n- I consider the first paragraph of Section 4 to be the \"meat\" of the paper, and I think some of this content should be front-loaded more.  It's a shame to wait until page 6.\n- I would start each of 4.1/2/3 with a paragraph giving a non-mathematical, qualitative description of the result.\n- I like the paragraph at top of page 7, which gives an explanation of how reward learning would act/fail.  Can you include such a paragraph in 4.1/3 as well?  I think the more you mirror structure in these sections, the better. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "0hIITL-hCcP", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "------------------------------------------------------------\nPost-rebuttal\n-----------------------------------------------------------\nGiven the effort of the authors of improving their manuscript, I am improving my original score. However, my evaluation is still \"weak reject\" for the reasons below:\n\n(1) I still fail to see clear differences between \"assistance\", as defined by the authors, and the other reinforcement learning-like approaches that assume that the reward function is unknown. I can see that they perhaps provide a more organized and methodological description of how that \"assistance\" can happen when compared to the previous works. However, the paper lacks practical advice, exactly how should I build an agent to leverage such \"assistance\"? I don't think their ideas are so novel that other methods couldn't be at least adapted to work in their scenario (to include some empirical evaluation in the manuscript).\n\n(2) The paper seems a little displaced to me in this conference. The paper neither provides practical and direct guidance on how to build algorithms to leverage \"assistance\", nor is a survey that focuses on organizing the area and discussing differences between works. Perhaps the paper would be better placed in a \"Blue Sky\" track. \n\n-----------------------------------------------------------\n\nThe authors propose two learning paradigms where the learning agent doesn't have access to a reward function but instead has to learn directly from the \"assistance\" from a trainer agent.\n\nWhile looking for ways to facilitate task specification and human integration in the learning process is a relevant and promising research goal, the authors don't explain the difference between their newly-proposed paradigms and the very rich literature on Inverse Reinforcement Learning and Learning from Demonstrations.\n\nLearning from human \"assistance\" instead of a reward function is not a new thing, and the surveys below (not cited by the authors) summarize a rich literature that does precisely that:\n\nSilva, Felipe Leno, and Anna Helena Reali Costa. \"A survey on transfer learning for multiagent reinforcement learning systems.\" Journal of Artificial Intelligence Research 64 (2019): 645-703. -> Surveys many categories of works where one agent provides guidance to others, including humans providing \"assistance\" to learning agents.\n\nArgall, Brenna D., et al. \"A survey of robot learning from demonstration.\" Robotics and autonomous systems 57.5 (2009): 469-483. -> Surveys learning from demonstration, where a human provides policy demonstrations to a learning agent, which usually doesn't have access to a reward function\n\nGao, Yang, et al. \"A survey of inverse reinforcement learning techniques.\" International Journal of Intelligent Computing and Cybernetics (2012). -> Inverse reinforcement learning, where the learning agent doesn't have access to a reward function and has to infer a policy from human assistance\n\nWithout a comprehensive discussion about the differences between the authors' proposal and those paradigms, I can't judge the paper contribution. To my eyes, all the descriptions gave in the paper look just like the same as one of the problems surveyed in the papers above.\n\nFor example, \"Non-active reward learning\" seems to me equivalent to inverse reinforcement learning.\n\nAlso, \"active reward learning\" seems to me a special case of the action advising problem surveyed in the first paper above.\n\nIn the case the authors deal with a different problem in this paper, I suggest that the manuscript is rewritten to thoughtfully explain the difference between all those scenarios. In case they indeed are correspondent scenarios, I suggest that the authors use the same notation as the previous works, and include comparisons with the state of the art methods in the experimental evaluation.\n\n-----------------------------------------------------\nOther suggestions\n\n- I don't get how the policy decision function will compute the expected reward if the reward function is unknown.\n\n- You assume access to a \"human decision function\", what exactly does that mean? do you need the probabilities for taking each action? If that's the case it is very unrealistic to expect that a human is able to provide probabilities for each action. Asking s/he to simply pick one action when requested is more realistic.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a well written paper, outlining a class of assistive algorithms. Being more or less a survey paper, it could do a better job of discussing 'inverse reinforcement learning' and 'collaborative inverse reinforcement learning'. It could also be slightly more general: for example the human dewcision function need not be known if we model the interaction as a Bayesian game (then the human might have a latent type, which can be inferred together with the reward function). The active reward learning problem is sometimes referred to as 'preference elicitation'. In the end, it was not clear that the discussion in this paper had any actionable insights for future models or algorithms in this area.

Decision: Reject