Information Directed Reward Learning for Reinforcement Learning

1 Introduction . Reinforcement learning ( RL ; Sutton and Barto , 2018 ) casts the problem of learning to perform complex tasks by interacting with an environment as an optimization problem where the learning agent aims to maximize its expected cumulative reward . Despite the remarkable successes of RL ( e.g. , Mnih et al. , 2015 ; Silver et al. , 2016 ) , specifying reward functions that capture complex tasks is still an open problem . A promising approach is to learn a reward function from human feedback ( e.g. , Christiano et al. , 2017 ) . However , since human feedback is expensive , active reward learning aims to minimize the number of queries . Prior work often focuses on approximating the reward function uniformly well . However , this may not be aligned with the original goal of RL : finding an optimal policy , as Figure 1 shows . Moreover , prior work is often tailored to ∗Work done while at Microsoft Research Cambridge . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . T = 4 π1 π2 Ĝ ( π1 ) = r̂ ( ) + r̂ ( ) Ĝ ( π2 ) = r̂ ( ) + r̂ ( ) Ĝ ( π1 ) − Ĝ ( π2 ) = r̂ ( ) − r̂ ( ) argmax q∈ { , , , } I ( Ĝ ( π1 ) − Ĝ ( π2 ) , ( q , ŷ ) ) ︸ ︷︷ ︸ IDRL objective = { , } Figure 1 : The robot wants to collect food for a human . It can only move 4 timesteps in the gridworld , can not pass through the black walls , and collecting more food is always better . The robot does not know the human ’ s preferences , but it can ask for food ratings . Common active learning methods aim to learn the reward uniformly well , and would query all items similarly often . In contrast , IDRL considers only the two plausibly optimal policies π1 and π2 . Since both policies collect the cherry , and do not collect the pear , the robot only needs to learn about the apple and the corn . IDRL can solve the task with 2 queries instead of 4. specific types of queries , such as comparisons of two trajectories ( e.g. , Sadigh et al. , 2017 ) or numerical evaluations of trajectories ( e.g. , Daniel et al. , 2015 ) , limiting its applicability . Contributions . We propose Information Directed Reward Learning ( IDRL ) , a general active reward learning approach for learning a model of the reward function from expensive feedback with the goal of finding a good policy rather than uniformly reducing the model ’ s error . IDRL can use arbitrary Bayesian reward models and arbitrary types of queries ( Section 4 ) , making it more general than existing methods . We describe an exact and efficient implementation of IDRL using Gaussian process ( GP ) reward models ( Section 5 ) and different types of queries , and an approximation of IDRL that uses a deep neural network reward model and a state-of-the-art policy gradient algorithm to learn from comparison queries ( Section 6 ) . We evaluate IDRL extensively in simulated environments ( Section 7 ) , including a driving task and high-dimensional continuous control tasks in the MuJoCo simulator , and show that both implementations significantly outperform prior methods . 2 Related work . Reward Learning for RL . Several works aim to directly learn policies rather than reward functions from expert feedback in the form of numerical evaluations ( Knox and Stone , 2009 ; MacGlashan et al. , 2017 ) or comparisons ( Regan and Boutilier , 2009 ; Fürnkranz et al. , 2012 ) , and some work also explores active query selection ( Akrour et al. , 2012 ; Wilson et al. , 2012 ) . However , learning policies directly from feedback has several downsides : it is difficult to combine different types of feedback , and policies tend to generalize poorly between environments . Ng and Russell ( 2000 ) argue that reward functions are a more robust representation of desired behavior than policies . Inverse reinforcement learning ( IRL ) aims to learn a reward model from expert demonstrations ( Abbeel and Ng , 2004 ) . Reward models can also be learned from comparisons of two or more different behaviors ( Wirth et al. , 2017 ) , or other kinds of feedback ( Jeon et al. , 2020 ) . While reward models can successfully learn hard-to-specify control and game-playing tasks ( Christiano et al. , 2017 ; Ibarz et al. , 2018 ) , most work uses simple heuristics to select queries to make . In contrast , active reward learning aims to select the most informative queries in a principled way . Active reward learning . For linear reward functions , Sadigh et al . ( 2017 ) ask the expert to compare trajectories synthesized to maximize the volume removed from a hypothesis space . Bıyık et al . ( 2020b ) argue that maximizing information gain leads to better sample efficiency and queries that are easier to answer than volume removal . Bıyık et al . ( 2020a ) generalize maximizing information gain to non-linear reward functions using a GP model . We also use an information gain objective to select queries ; however , our approach focuses on finding an optimal policy instead of uniformly reducing the error of the reward model . With a similar motivation , Wilde et al . ( 2020 ) aim to capture how informative a query is for distinguishing policies . However , their method is limited to comparisons between potentially optimal policies . Daniel et al . ( 2015 ) also introduce an acquisition function to measure how informative a query is for learning a good policy . However , their setting is restricted to observing the cumulative reward of a trajectory , and their acquisition function is computationally expensive . Table 1 gives an overview of how our method compares to this prior work . Bayesian optimization . Bayesian optimization ( BO ) aims to maximize an expensive-to-evaluate function by learning a Bayesian model of the function and selecting informative queries ( Mockus et al. , 1978 ) . We face a related but significantly harder problem : we aim to find an optimal policy in RL but only indirectly obtain information about the value of policies . Our problem has striking connections to variants of the multi-armed bandit problem ( Bubeck and Cesa-Bianchi , 2012 ) , in particular partial monitoring problems ( Rustichini , 1999 ) and transductive linear bandits ( Fiez et al. , 2019 ) . We explore this connection in detail in Appendix C . 3 Background and problem setting . Markov decision process . Markov decision processes ( MDPs ; Puterman , 2014 ) model sequential decision-making problems in dynamical systems . An MDP ( S , A , P , r , p0 , γ ) consists of a state space S , an action space A , a transition function P , a reward function r , an initial state distribution p0 , and a discount factor γ ∈ [ 0 , 1 ) . In an MDP , the agent starts in state s0 ∼ p0 ( s ) and , when taking action at , transitions from state st to state st+1 with probability P ( st+1|st , at ) . The agent affects the environment through actions determined by a policy π ( at|st ) , indicating the probability of taking action at in state st . The agent ’ s goal is to find a policy π that maximizes the expected discounted return G ( π ) = EP , π , p0 [ ∑∞ t=0 γ trt ] , where rt is the reward obtained at time t. Information gain . Intuitively , the information gain between two random variables measures the amount of information that can be obtained about one of them by observing the other . Formally , for two random variables X and Y with marginal distributions pX , pY and joint distribution p ( X , Y ) , the information gain ( or mutual information ) is I ( X , Y ) = DKL ( p ( X , Y ) ‖pX ·pY ) , whereDKL ( ·‖· ) is the KL-divergence . Given a third random variable Z , conditional information gain is defined as I ( X ; Y |Z = z ) = DKL ( p ( X , Y ) |Z=z‖pX|Z=z · pY |Z=z ) . Problem setting . We focus on MDPs where the reward function is not readily available . Instead , the agent can query an expert for information about the reward . In iteration i , the agent makes a query qi to the expert , and receives a response yi . For example , qi could ask the expert to compare two trajectories or judge a single trajectory , and yi could indicate which of the two trajectories is better or provide the return of a single trajectory . We assume that the agent can interact with the environment cheaply , but queries to the expert are expensive , and hence the agent has to find a policy π that maximizes the expected return G ( π ) using as few queries as possible . Our reward learning approach . We approach this problem by learning a model of the reward function , i.e. , a model that predicts the reward of a given state,2 and computing a policy that maximizes the return induced by the model . Importantly , we want to learn a reward model such that the induced optimal policy achieves a high return under the true reward function . Note that any RL algorithm can be used to find the policy . Hence , the problem reduces to selecting a model for the reward function and deciding which queries to make . Our key insight is that queries that help most to find a good policy might differ from those that uniformly reduce the model ’ s uncertainty . 2Our approach is also applicable to reward functions that depend on state-action pairs or transitions . We focus on state-dependent reward functions for simplicity of exposition . Algorithm 1 Information Directed Reward Learning ( IDRL ) . The algorithm requires a set of candidate queriesQc , a Bayesian model of the reward function , and an RL algorithm that returns a policy given a reward function . Ĝ ( π ) is the belief about the expected return of policy π , induced by the reward model P ( r̂|D ) , and r̂ is the belief about the reward function . 1 : D ← { } ; Πc ← initialize candidate policies ; initialize reward model with prior distribution P ( r̂ ) 2 : while not converged do 3 : Select a query : 4 : π1 , π2 ∈ argmaxπ , π′∈Πc H ( Ĝ ( π ) − Ĝ ( π ′ ) |D ) 5 : q∗ ∈ argmaxq∈Qc I ( Ĝ ( π1 ) − Ĝ ( π2 ) ; ( q , ŷ ) |D ) 6 : Make query and update reward model : 7 : y∗ ← Response to query q∗ 8 : P ( r̂|D ∪ { ( q∗ , y∗ ) } ) ∝ P ( y∗|r̂ , D , q∗ ) P ( r̂|D ) . Update belief about the reward 9 : D ← D ∪ { ( q∗ , y∗ ) } . Add observation to dataset 10 : Optionally update candidate policies Πc 11 : end while 12 : r̄ ← mean estimate of the reward model ; π̄∗ ← RL ( r̄ ) 13 : return π̄∗

Summary: Information Directed Reward Learning (IDRL) actively selects queries that are most informative at distinguishing policies in terms of their value. By contrast, prior work has typically tried to maximize information gained about the reward function itself -- but some parts of the state space may be unreachable, or never visited by any plausible optimal policy, and so learning about them is irrelevant for task performance. IDRL has a pleasing theoretical motivation, although practical implementations of it require considerable use of approximations and heuristics. Additionally, IDRL is compatible with a broad variety of reward learning queries (e.g. preference comparisons, labeling trajectories with returns) whereas most prior work focuses on a single feedback modality. Experimental results show significant improvements in sample efficiency from use of IDRL relative to a variety of baselines.