The impacts of known and unknown demonstrator irrationality on reward inference

1 INTRODUCTION . Motivated by difficulty in reward specification ( Lehman et al. , 2018 ) , inverse reinforcement learning ( IRL ) methods estimate a reward function from human demonstrations ( Ng et al. , 2000 ; Abbeel and Ng , 2004 ; Kalman , 1964 ; Jameson and Kreindler , 1973 ; Mombaur et al. , 2010 ) . The central assumption behind these methods is that human behavior is rational , i.e. , optimal with respect to their reward ( cumulative , in expectation ) . Unfortunately , decades of research in behavioral economics and cognitive science ( Chipman , 2014 ) have unearthed a deluge of irrationalities , i.e. , of ways in which people deviate from optimal decision making : hyperbolic discounting , scope insensitivity , illusion of control , decision noise , loss aversion , to name a few . While as a community we are starting to account for some possible irrationalities plaguing demonstrations in different ways ( Ziebart et al. , 2008 ; 2010 ; Singh et al. , 2017 ; Reddy et al. , 2018 ; Evans et al. , 2016 ; Shah et al. , 2019 ) , we understand relatively little about what effect irrationalities have on the difficulty of inferring the reward . In this work , we seek a systematic analysis of this effect . Do irrationalities make it harder to infer the reward ? Is it the case that the more irrational someone is , the harder it is to infer the reward ? Do we need to account for the specific irrationality type during learning , or can we get away with the standard noisy-rationality model ? The answers to these questions are important in deciding how to move forward with reward inference . If irrationality , even when well modelled , still makes reward inference very difficult , then we will need alternate ways to specify behaviors . If well-modelled irrationality leads to decent reward inference but we run into problems when we just make a noisy-rational assumption , that suggests we need to start accounting for irrationality more explicitly , or at least seek assumptions or models that are robust to many different types of biases people might present . Finally , if the noisy-rational model leads to decent inference even when the demonstrator is irrational , then we need not dedicate significant resources to addressing irrationality . One challenge with conducting such an analysis is that there are many irrationalities in the psychology and behavioral economics literature , with varying degrees of mathematical formalization versus empirical description . To structure the space for our analysis , we operationalize irrationalities in the language of MDPs by systematically enumerating possible deviations from the Bellman equation – imperfect maximization , deviations from the true transition function , etc . This gives us a formal framework in which we can simulate irrational behavior , run reward inference , and study its performance . Armed with this formalism , we then explore the various impacts of irrationality on reward learning in three families of environments : small random MDPs , a more legible gridworld MDP , and an autonomous driving domain drawn from the robotics literature ( Sadigh et al. , 2016 ) . Irrationality can help , rather than hinder reward inference – if it is modelled correctly . We first explore the impacts of demonstrator irrationality when the irrationality is known to the reward inference algorithm . Surprisingly , we find that certain irrationalities actually improve the quality of reward inference - that is , they make reward easier to learn . Importantly , this is not compared to assuming the wrong model of the human : our finding is that humans who exhibit ( correctly modelled ) irrationality are more informative than humans who exhibit ( correctly modelled ) rationality ! This is consistent in all three domains . We explain this theoretically from the perspective of the mutual information between the demonstrator behavior and the reward parameters , proving that some irrationalities are arbitrarily more informative than rational behavior . Unmodelled irrationality leads to remarkably poor reward inference . It might seem that we can ’ t immediately benefit from the knowledge that irrationalities help inference unless we have a comprehensive understanding of human decision-making , and so we should just stick to the status quo of modeling people as rational . However , we find that modeling irrational demonstrators as noisily-rational can lead to worse outcomes than not performing inference at all and just using the prior ( section 5 ) . Encouragingly , we also find evidence that even just modeling the demonstrator ’ s irrationality approximately allows a learner to outperform modeling the demonstrator as noisilyrational ( section E ) . Overall , we contribute 1 ) a theoretical and empirical analysis of the effects of different irrationalities on reward inference , 2 ) a way to systematically formalize and cover the space of irrationalities in order to conduct such an analysis , and 3 ) evidence for the importance and benefit of accounting for irrationality irrationality during inference . Our results suggest that modeling people as noisily rational leads to poor reward inference , and that it is important to model the irrationalities of human demonstrators . Our good news is that if we manage to do that well , we might be better off even compared to a counterfactual world in which people are actually rational ! Of course , modeling irrationality is a long term endeavour . Our near-term good news is two fold : first , irrationalities can be an ally for teaching . For example , we could ask human demonstrators to act more myopically to better communicate their reward to the learners . Second , we need not get the biases perfectly correct to do better than assuming noisy-rationality . Instead , using slightly more realistic models of human irrationality could lead to better inference . 2 FRAMEWORK : BIASES AS DEVIATIONS FROM THE BELLMAN UPDATE . 2.1 EXPLORING BIASES THROUGH SIMULATION . While ideally we would recruit human subjects with different irrationalities and measure how well we can learn rewards , this is prohibitive because we do not get to dictate someone ’ s irrationality type : people exhibit a mix of them , some yet to be discovered . Further , measuring the accuracy of inference from observing real humans is complicated by the fact that we do not have ground truth access to the human ’ s reward function . For instance , suppose we asked subjects to produce a set of ( behavior , reward function ) pairs . We could then try to predict the reward functions from the behaviors . But how did we , the experimenters , infer the reward functions from the people ? If we are wrong in our assumptions about which irrationalities are affecting their behavior and/or explicit reports of rewards , we would remain deluded about the subjects ’ true intentions and preferences . To address these issues , we simulate demonstrator behavior subject to different irrationalities , run reward inference , and measure the performance against the ground truth , i.e. , the accuracy of a Bayesian posterior on the reward parameter given the ( simulated ) demonstrator ’ s inputs . 2.2 BACKGROUND AND FORMALISM . Consider an Uncertain-Reward MDP ( URMDP ) ( Bagnell et al. , 2001 ; Regan and Boutilier , 2011 ; Desai , 2017 ) M = ( S , A , { Ps , a } , γ , Θ , p , r ) , consisting of finite state and action sets S and A , distributions over states { Ps , a } representing the result of taking action a in state s , discount rate γ ∈ [ 0 , 1 ) , a ( finite ) set of reward parameters Θ , a prior distribution p ∈ ∆ ( S×Θ ) over starting states and reward parameters , and a parameterized state-action reward function r : Θ× S ×A× S → R , where rθ ( s , a , s′ ) represents the reward received . We assume that the human demonstrator ’ s policy π satisfies π = d ( θ ) , where d is an ( environmentspecific ) planner d : Θ → Π that returns a ( possibly stochastic ) policy given a particular reward parameter θ . The rational demonstrator uses a planner dRational that , given a reward parameter θ , returns a policy that maximizes its expected value . On the other hand , we say that an demonstrator is irrational if its planner returns policies with lower expected value than the optimal policy , for at least one θ ∈ Θ . 2.3 TYPES AND DEGREES OF IRRATIONALITY . There are many possible irrationalities that people exhibit ( Chipman , 2014 ) , far more than what we could study in one paper . To provide good coverage of this space , we start from the Bellman update , and systematically manipulate its terms and operators to produce a variety of different irrationalities that deviate from the optimal MDP policy in complementary ways . For instance , operating on the discount factor can model myopic behavior , while operating on the transition function can model optimism or the illusion of control . We parametrize each irrationality so that we can manipulate its “ intensity '' or deviation from rationality . Figure 2 summarizes our approach , which we detail below . 2.3.1 RATIONAL DEMONSTRATOR . In our setup , the rational demonstrator does value iteration using the Bellman update from Fig . 2 . Our models change this update to produce different types of non-rational behavior . 2.3.2 MODIFYING THE MAX OPERATOR : BOLTZMANN . Boltzmann-rationality modifies the maximum over actions maxa with a Boltzmann operator with a parameter β : Vi+1 ( s ) = Boltzβa ∑ s′∈S Ps , a ( s ′ ) ( rθ ( s , a , s ′ ) + γVi ( s ′ ) ) , where Boltzβ ( x ) = ∑ i xie βxi/ ∑ i e βxi ( Ziebart et al. , 2010 ; Asadi and Littman , 2017 ) . This is the most popular stochastic model used in reward inference ( Ziebart et al. , 2010 ; Asadi and Littman , 2017 ; Fisac et al. , 2017 ) . After computing the value function , the Boltzmann-rational planner dBoltz returns a policy where the probability of an action is proportional to the exponential of the Q-value of the action : π ( a|s ) ∝ eβQθ ( s , a ) . The constant β is called the rationality constant , because as β →∞ , the human choices approach perfect rationality ( optimality ) , whereas β = 0 produces uniformly random choices . 2.3.3 MODIFYING THE TRANSITION FUNCTION . Our next set of irrationalities manipulate the transition function away from reality . Illusion of Control . People often overestimate their ability to control random events ( Thompson , 1999 ) . To model this , we consider demonstrators that use the Bellman update : Vi+1 ( s ) = max a ∑ s′∈S Pns , a ( s ′ ) ( rθ ( s , a , s ′ ) + γVi ( s ′ ) ) where Pns , a ( s ′ ) ∝ ( Ps , a ( s′ ) ) n. As n → ∞ , the demonstrator acts as if it exists in a deterministic environment . As n→ 0 , the demonstrator acts as if it had an equal chance of transitioning to every possible successor state . Optimism/Pessimism . Many people systematically overestimate or underestimate their chance experiencing of positive over negative events ( Sharot et al. , 2007 ) . We model this using demonstrators that modify the probability they get outcomes based on the value of those outcomes : Vi+1 ( s ) = max a ∑ s′∈S Pωs , a ( s ′ ) ( rθ ( s , a , s ′ ) + γVi ( s ′ ) ) where Pωs , a ( s ′ ) ∝ Ps , a ( s′ ) eω ( rθ ( s , a , s ′ ) +γVi ( s ) ) . ω controls how pessimistic or optimistic the demonstrator is . As ω → +∞ ( respectively , ω → −∞ ) , the demonstrator becomes increasingly certain that good ( bad ) transitions will happen . As ω → 0 , the demonstrator approaches the rational demonstrator .

Summary: This work studies the effect of modeling systematic irrationality of the demonstrator for reward learning problems. By manipulating different factors of Bellman update, the authors simulate different irrational behavior in demonstrations. Experiments in gridworld and a 2D driving domain demonstrate that modeling irrationality helps reward inference. The authors also demonstrate that knowing the general/approximate type of irrationality, instead of knowing the exact irrationality model, might be enough to improve reward inference.