Benefits of Assistance over Reward Learning

1 INTRODUCTION . Traditional computer programs are instructions on how to perform a particular task . However , we do not know how to mechanically perform more challenging tasks like translation . The field of artificial intelligence raises the level of abstraction so that we simply specify what the task is , and let the machine to figure out how to do it . As task complexity increases , even specifying the task becomes difficult . Several criteria that we might have thought were part of a specification of fairness turn out to be provably impossible to simultaneously satisfy ( Kleinberg et al. , 2016 ; Chouldechova , 2017 ; Corbett-Davies et al. , 2017 ) . Reinforcement learning agents often “ game ” their reward function by finding solutions that technically achieve high reward without doing what the designer intended ( Lehman et al. , 2018 ; Krakovna , 2018 ; Clark & Amodei , 2016 ) . In complex environments , we need to specify what not to change ( McCarthy & Hayes , 1981 ) ; failure to do so can lead to negative side effects ( Amodei et al. , 2016 ) . Powerful agents with poor specifications may pursue instrumental subgoals ( Bostrom , 2014 ; Omohundro , 2008 ) such as resisting shutdown and accumulating resources and power ( Turner , 2019 ) . A natural solution is to once again raise the level of abstraction , and create an agent that is uncertain about the objective and infers it from human feedback , rather than directly specifying some particular task ( s ) . Rather than using the current model of intelligent agents optimizing for their objectives , we would now have beneficial agents optimizing for our objectives ( Russell , 2019 ) . Reward learning ( Leike et al. , 2018 ; Jeon et al. , 2020 ; Christiano et al. , 2017 ; Ziebart et al. , 2010 ) attempts to instantiate this by learning a reward model from human feedback , and then using a control algorithm to optimize the learned reward . Crucially , the control algorithm does not reason about the effects of the chosen actions on the reward learning process , which is external to the environment . In contrast , in the assistance paradigm ( Hadfield-Menell et al. , 2016 ; Fern et al. , 2014 ) , the human H is modeled as part of the environment and as having some latent goal that the agent R ( for robot ) does not know . R ’ s goal is to maximize this ( unknown ) human goal . In this formulation , R must balance between actions that help learn about the unknown goal , and control actions that lead to high reward . Our key insight is that by integrating reward learning and control modules , assistive agents can take into account the reward learning process when selecting actions . This gives assistive agents a significant advantage over reward learning agents , which can not perform similar reasoning . The goal of this paper is to clarify and illustrate this advantage . We first precisely characterize the differences between reward learning and assistance , by showing that two phase , communicative assistance is equivalent to reward learning ( Section 3 ) . We then give qualitative examples of desirable behaviors that can only be expressed once these restrictions are lifted , and thus are only exhibited by assistive agents ( Section 4 ) . Consider for example the kitchen environment illustrated in Figure 1 , in which R must bake a pie for H . R is uncertain about which type of pie H prefers to have , and currently H is at work and can not answer R ’ s questions . An assistive R can make the pie crust , but wait to ask H about her preferences over the filling ( Section 4.1 ) . R may never clarify all of H ’ s preferences : for example , R only needs to know how to dispose of food if it turns out that the ingredients have gone bad ( Section 4.2 ) . If H will help with making the pie , R can allow H to disambiguate her desired pie by watching what filling she chooses ( Section 4.3 ) . Vanilla reward learning agents do not show these behaviors . We do not mean to suggest that all work on reward learning should cease and only research on assistive agents should be pursued . Amongst other limitations , assistive agents are very computationally complex . Our goal is simply to clarify what qualitative benefits an assistive formulation could theoretically provide . Further research is needed to develop efficient algorithms that can capture these benefits . Such algorithms may look like algorithms designed to solve assistance problems as we have formalized them here , but they may also look like modified variants of reward learning , where the modifications are designed to provide the qualitative benefits we identify . 2 BACKGROUND AND RELATED WORK . We introduce the key ideas behind reward learning and assistance . X∗ denotes a sequence of X . We use parametric specifications for ease of exposition , but our results apply more generally . 2.1 POMDPS . A partially observable Markov decision process ( POMDP ) M = 〈S , A , Ω , O , T , r , P0 , γ〉 consists of a finite state space S , a finite action space A , a finite observation space Ω , an observation function O : S → ∆ ( Ω ) ( where ∆ ( X ) is the set of probability distributions over X ) , a transition function T : S × A → ∆ ( S ) , a reward function r : S × A × S → R , an initial state distribution P0 : ∆ ( S ) , and a discount rate γ ∈ ( 0 , 1 ) . We will write ot to signify the tth observation O ( st ) . A solution to the POMDP is given by a policy π : ( O×A ) ∗×O → ∆ ( A ) that maximizes the expected sum of rewards ER ( π ) = Es0∼P0 , at∼π ( ·|o0 : t , a0 : t−1 ) , st+1∼T ( ·|st , at ) [ ∑∞ t=0 γ tr ( st , at , st+1 ) ] . 2.2 REWARD LEARNING . We consider two variants of reward learning : non-active reward learning , in which R must infer the reward by observing H ’ s behavior , and active reward learning , in which R may choose particular questions to ask H in order to get particular feedback . A non-active reward learning problem P = 〈M\r , C , 〈Θ , rθ , PΘ〉 , πH , k〉 contains a POMDP without rewardM\r = 〈S , AR , ΩR , OR , T , P0 , γ〉 , and instead R has access to a parameterized reward space 〈Θ , rθ , PΘ〉 . R is able to learn about θ∗ by observing H make k different choices c , each chosen from a set of potential choices C. In order for R to learn from the human ’ s choices , it also assumes access to the human decision function πH ( c | θ ) that determines how the human makes choices for different possible reward functions rθ . Common decision functions include perfect optimality ( Ng & Russell , 2000 ) and Boltzmann rationality ( Ziebart et al. , 2010 ) . There are many types of choices ( Jeon et al. , 2020 ) , including demonstrations ( Argall et al. , 2009 ; Ng & Russell , 2000 ; Ziebart et al. , 2010 ; Fu et al. , 2017 ; Gao et al. , 2012 ) , comparisons ( Zhang et al. , 2017 ; Wirth et al. , 2017 ; Christiano et al. , 2017 ; Sadigh et al. , 2017 ) , corrections ( Bajcsy et al. , 2017 ) , the state of the world ( Shah et al. , 2019 ) , proxy rewards ( Hadfield-Menell et al. , 2017b ) , natural language ( Fu et al. , 2019 ) , etc . A policy decision function f ( c0 : k−1 ) produces a policy πR after observing H ’ s choices . A solution is a policy decision function f that maximizes expected reward Eθ∼PΘ , c0 : k−1∼πH [ ER ( f ( c0 : k−1 ) ) ] . Since H ’ s choices c0 : k−1 do not affect the state of the environment that R is acting in , this is equivalent to choosing πR that maximizes expected reward given the posterior over reward functions , that is Eθ∼P ( θ|c0 : k−1 ) [ ER ( πR ) ] . An active reward learning problem P = 〈M\r , Q , C , 〈Θ , rθ , PΘ〉 , πH , k〉 adds the ability forR to ask H particular questions q ∈ Q in order to get more targeted feedback about θ . The human decision function πH ( c | q , θ ) now depends on the question asked . A solution consists of a question policy πRQ ( qi | q0 : i−1 , c0 : i−1 ) and a policy decision function f ( q0 : k−1 , c0 : k−1 ) that maximize expected reward Eθ∼PΘ , q0 : k−1∼πRQ , c0 : k−1∼πH [ ER ( f ( q0 : k−1 , c0 : k−1 ) ) ] . A typical algorithm ( Eric et al. , 2008 ; Daniel et al. , 2014 ; Maystre & Grossglauser , 2017 ; Christiano et al. , 2017 ; Sadigh et al. , 2017 ; Zhang et al. , 2017 ; Wilde et al. , 2020 ) will compute and ask q ∈ Q that maximizes an active learning criterion such as information gain ( Bıyık et al. , 2019 ) or volume removal ( Sadigh et al. , 2017 ) . Best results are achieved by selecting questions with the highest value of information ( Cohn , Robert W , 2016 ; Zhang et al. , 2017 ; Mindermann et al. , 2018 ; Wilde et al. , 2020 ) , but these are usually much more computationally expensive . R then finds a policy that maximizes expected reward under the inferred distribution over θ , in order to approximately solve the original POMDP . Note that a non-active reward learning problem is equivalent to an active reward learning problem with only one question , since having just a single question means that R has no choice in what feedback to get ( see Appendix A.1 for proofs ) . 2.3 ASSISTANCE . The key idea of assistance is that helpful behaviors like reward learning are incentivized when R does not know the true reward r and can only learn about it by observing human behavior . So , we model the human H as part of the environment , leading to a two-agent POMDP , and assume there is some true reward r that only H has access to , while the robot R only has access to a model relating r to H ’ s behavior . Intuitively , as R acts in the environment , it will also observe H ’ s behavior , which it can use to make inferences about the true reward . Following Hadfield-Menell et al . ( 2016 ) 1 , we define an assistance gameM as a tuple M = 〈S , { AH , AR } , { ΩH , ΩR } , { OH , OR } , T , PS , γ , 〈Θ , rθ , PΘ〉〉 . Here S is a finite set of states , AH a finite set of actions for H , ΩH a finite set of observations for H , and OH : S → ∆ ( ΩH ) an observation function for H ( respectively AR , ΩR , OR for R ) . The transition function T : S×AH ×AR → ∆ ( S ) gives the probability over next states given the current state and both actions . The initial state is sampled from PS ∈ ∆ ( S ) . Θ is a set of possible reward function parameters θ which parameterize a class of reward functions rθ : S ×AH ×AR × S → R , and Pθ is the distribution from which θ is sampled . γ ∈ ( 0 , 1 ) is a discount factor . As with POMDPs , policies can depend on history . Both H and R are able to observe each other ’ s actions , and on a given timestep , R acts before H . We use τRt : ( Ω R × AH × AR ) t to denote 1Relative to Hadfield-Menell et al . ( 2016 ) , our definition allows for partial observability and requires that the initial distribution over S and Θ be independent . We also have H choose her action sequentially after R , rather than simultaneously with R , in order to better parallel the reward learning setting . R ’ s observations until time t , and τHt for H ’ s observations ; thus R ’ s policy can be written as πR ( aR | oRt , τRt−1 ) , while H ’ s can be written as πH ( aH | oHt , aRt τHt−1 , θ ) . Note that unlike H , R does not observe the reward parameter θ , and must infer θ much like it does the hidden state . A fully observable assistance game is one in which both H and R can observe the full state . In such cases , we omit ΩH , ΩR , OH and OR . Since we have not yet specified how H behaves , it is not clear what the agent should optimize for . Should it be playing a Nash strategy or optimal strategy pair of the game , and if so , which one ? Should it use a non-equilibrium policy , since humans likely do not use equilibrium strategies ? This is a key hyperparameter in assistance games , as it determines the communication protocol for H and R. For maximum generality , we can equip the assistance game with a policy-conditioned belief B : ΠR → ∆ ( ΠH ) over πH , which specifies how the human responds to the agent ’ s choice of policy ( Halpern & Pass , 2018 ) . The agent ’ s goal is to maximize expected reward given this belief . Prior work on assistance games ( Hadfield-Menell et al. , 2016 ; Malik et al. , 2018 ; Woodward et al. , 2019 ) focuses on finding optimal strategy pairs . This corresponds to a belief thatH will know and perfectly respond toR ’ s policy ( see Appendix A.3 ) . However , our goal is to compare assistance to reward learning . Typical reward learning algorithms assume access to a model of human decision-making : for example , H might be modeled as optimal ( Ng & Russell , 2000 ) or Boltzmann-rational ( Ziebart et al. , 2010 ) . As a result , we also assume that we have access to a model of human decision-making πH . Note that πH depends on θ : we are effectively assuming that we know how H chooses how to behave given a particular reward rθ . This assumption corresponds to the policy-conditioned belief B ( πR ) ( π̃ H ) = 1 [ π̃H = πH ] . We define an assistance problem P as a pair 〈M , πH〉 where πH is a human policy for the assistance gameM . Given an assistance problem , a robot policy πR induces a probability distribution over trajectories : τ ∼ 〈s0 , θ , πH , πR〉 , τ ∈ [ S ×AH ×AR ] ∗ . We denote the support of this distribution by Traj ( πR ) . The expected reward of a robot policy for 〈M , πH〉 is given by ER ( πR ) = E s0∼PS , θ∼Pθ , τ∼〈s0 , θ , πH , πR〉 [ ∞∑ t=0 γtrθ ( st , a H t , a R t , st+1 ) ] . A solution of 〈M , πH〉 is a robot policy that maximizes expected reward : πR = argmax π̃R ER ( π̃R ) .

Summary: This work proposes learning a single control policy for human-in-the-loop learning rather than having a reward learning component and a control component. The key difference is that the action selection can use information from the reward learning module. The authors formulate an assistance game in this setting and show that it can reduce to an equivalent POMDP. The work then describes a communicative assistance problem and shows the equivalence of reward learning to assistance and visa versa. Results show qualitative improvements on variants of the kitchen domain.