The impacts of known and unknown demonstrator irrationality on reward inference
Authors: Lawrence Chan
Andrew Critch
Anca Dragan
Venue:  (2021)
Keywords: irrationality
reward learning
irl

Abstract: Algorithms inferring rewards from human behavior typically assume that people are (approximately) rational. In reality, people exhibit a wide array of irrationalities. Motivated by understanding the benefits of modeling these irrationalities, we analyze the effects that demonstrator irrationality has on reward inference. We propose operationalizing several forms of irrationality in the language of MDPs, by altering the Bellman optimality equation, and use this framework to study how these alterations affect inference. 

We find that incorrectly assuming noisy-rationality for an irrational demonstrator can lead to remarkably poor reward inference accuracy, even in situations where inference with the correct model leads to good inference. This suggests a need to either model irrationalities or find reward inference algorithms that are more robust to misspecification of the demonstrator model. Surprisingly, we find that if we give the learner access to the correct model of the demonstrator's irrationality, these irrationalities can actually help reward inference. In other words, if we could choose between a world where humans were perfectly rational and the current world where humans have systematic biases, the current world might counter-intuitively be preferable for reward inference. We reproduce this effect in several domains. While this finding is mainly conceptual, it is perhaps actionable as well: we might ask human demonstrators for myopic demonstrations instead of optimal ones, as they are more informative for the learner and might be easier for a human to generate.

Under review as a conference paper at ICLR 2021
THE IMPACTS OF KNOWN AND UNKNOWN DEMON- STRATOR IRRATIONALITY ON REWARD INFERENCE
Anonymous authors Paper under double-blind review
ABSTRACT
Algorithms inferring rewards from human behavior typically assume that people are (upproximately) rational. In reality, people exhibit a wide array of irrationalities. Motivated by understanding the benefits of modeling these irrationalities, we analyze the effects that demonstrator irrationality has on reward inference. We propose operationalizing several forms of irrationality in the language of MDPs, by altering the Bellman optimality equation, and use this framework to study how these alterations affect inference.
We find that incorrectly assuming noisy-rationality for an irrational demonstrator can lead to remarkably poor reward inference accuracy, even in situations where inference with the correct model leads to good inference. This suggests a need either model irrationalities or find reward inference algorithms that are more robust to misspecification of the demonstrator model. Surprisingly, we find that if we give the learner access the correct model of the demonstrator's 's irrationality, these irrationalities can actually help reward inference. In other words, if we could choose between a world where humans were perfectly rational and the current world where humans have systematic biases, the current world might counter- intuitively be preferable for reward inference. We reproduce this effect in several domains. While this finding is mainly conceptual, it is perhaps actionable well: we might ask human demonstrators for myopic demonstrations instead of optimal ones, they are more informative for the learner and might be easier for human to generate.
INTRODUCTION
Motivated by difficulty in reward specification (Lehman et al. 2018). inverse reinforcement learning (IRL) methods estimate reward function from human demonstrations (Ng et al.) 2000 Abbeel
and Ng 2004 Kalman 1964 Jameson and Kreindler 1973 Mombaur et al.) 2010). The central assumption behind these methods is that human behavior is rational, i.e., optimal with respect to their reward (cumulative, in expectation). Unfortunately, decades of research in behavioral economics and cognitive science (Chipman 2014) have unearthed deluge of irrationalities, i.e., of ways in which people deviate from optimal decision making: hyperbolic discounting, scope insensitivity, illusion of control, decision noise, loss aversion, name a few.
While as a community we are starting to account for some possible irrationalities plaguing demon- strations in different ways (Ziebart al., 2008 2010; Singh et al.) 2017; Reddy al. 2018 Evans et al. 2016 Shah et al., 2019). we understand rel relatively little about what effect irrationalities have on the difficulty of inferring the reward. In this work, we seek a systematic analysis of this effect. Do irrationalities make it harder to infer the reward? Is it the case that the more irrational someone is, the harder it is to infer the reward? Do we need to account for the specific irrationality type during learning, or can we get away with the standard noisy-rationality model? The answers to these questions are important in deciding how to move forward with reward inference. If irrationality, even when well modelled, still makes reward inference very difficult, then we will need alternate ways to specify behaviors. If well- modelled irrationality leads to decent reward inference but we run into problems when we just make a noisy-rations assumption, that suggests need to start accounting for irrationality more explicitly, or at least seek assumptions or models that are robust to many different types of biases people might present. Finally, if the noisy-rational model leads

---

Under review as a conference paper at ICLR 2021
Long Planning Horizon
Short Planning Horizon
Queed
High Wped
Low 6 opeed
High Opeed
Figure 1: We found that irrationality does not always hinder reward inference (section 3.2 is in many cases actually helpful. Here, we depict rational and myopic (short-sighted) behavior in merging environment (section (42)) for two different rewards, with higher and lower weights on going fast. The rational car (white) exhibits similar behavior under both reward functions, while the myopic car (blue) overtakes on the shoulder when the reward function places high weight on speed. This makes easier differentiate what its reward is.
to decent inference even when the demonstrator irrational, then we need not dedicate significant resources to addressing irrationality.
One challenge with conducting such an analysis that there are many irrationalities the psychology and behavioral economics literature, with varying degrees of mathematical formalization versus empirical description. To structure the space for our analysis, we operationalize irrationalities in the language of MDPs by systematically enumerating possible deviations from the Bellman equation - imperfect maximization, deviations from the true transition function, etc. This gives us a formal framework in which performance. Armed with this formalism, we then explore the various impacts of irrationality on reward learning in three families of environments: small random MDPs, more legible gridworld MDP, and an autonomous driving domain drawn from the robotics literature (Sadigh et al. 2016).
simulate irrational behavior, reward inference, and study its
Irrationality can help, rather than hinder reward inference if it is modelled correctly. We first explore the impacts of demonstrator irrationality when the irrationality is known to the reward inference algorithm. Surprisingly. we find that certain irrationalities actually improve the quality of reward inference- that is, they make reward easier to leamn Importantly, this is not compared to assuming the wrong model of the human: our finding is that humans who exhibit (correctly modelled) irrationality are more informative than humans who exhibit (correctly modelled) rationality! This consistent all three domains. We explain this theoretically from the perspective of the mutual information between the demonstrator behavior and the reward parameters, proving that some irrationalitics are arbitrarily more informative than rational behavior.
Unmodelled irrationality leads to remarkably poor reward inference. It might seem that can't immediately benefit from the knowledge that irrationalities help inference unless we have a comprehensive understanding of human decision-male making, and so should just stick to the status quo of modeling people as rational. However, we find that modeling irrational demonstrators as noisily-rational lead to worse outcomes than not performing inference all and just using the prior (section 5). Encouragingly, we also find evidence that even just modeling the demonstrator's s irrationality approximately allows a learner outperform modeling the demonstrator noisily- rational (section D)
Overall, contribute 1)x a theoretical and empirical analysis of the effects of different irrationalities on reward inference, 2) way to systematically formalize and cover the space of irrationalities in order conduct such an analysis, and 3) evidence for the importance and benefit of accounting for irrationality irrationality during inference.
Our results suggest that modeling people as noisily rational leads poor reward inference, and that is important to model the irrationalities of human demonstrators. Our good news is that if we manage to do that well, we might be better off even compared to a counterfactual world in which people are actually rational! Of course, modeling irrationality is long term endeavour. Our near-term good news is two fold: first, irrationalities can be an ally for teaching. For example, we could ask human demonstrators act more myopically to better communicate their reward to the learners. Second, need not get the biases perfectly correct to do better than assuming noisy- riisy-rationaltt.. Instead, using slightly more realistic models of human irrationality could lead to better inference.
2

---

Under review as a conference paper at ICLR 2021
Myspic Discounning
Myople Hostzon
Beltzmann-Rationall Rationallty
Prospect Blas
Hyperbolic Discounting
max (5+
Σ
Pa (ro(s, 8') +Vi(s'))
a
E'ES
Optimism Pessimism Illusion of Control
Extremal Blas
Figure 2: In section 2.3]
modify the components of the Bellman update to cover different types of irrationalities: changing the max into a softmax to capture noise (Bolizmann), changing the transition function capture optimism/pessimisnn or the illusion of control, changing the reward values to capture the nonlinear perception of gains and losses (Prospect), changing the average reward over time into a maximum (Extremal), and changing the discounting to capture more myopic decision-makee
FRAMEWORK: BIASES AS DEVIATIONS FROM THE BELLMAN UPDATE
2.1 EXPLORING BIASES THROUGH SIMULATION
While ideally we would recruit human subjects with different irrationalities and measure how well we can learn rewards, this is prohibitive because we do not get to dictate someone's 's irrationality type: people exhibit mix of them, some yet to be discovered. Further, measuring the accuracy of inference from observing real humans is complicated by the fact that we do not have ground truth access to the human's reward function. For instance, suppose asked subjects produce set of (behavior, reward function) pairs. We could then try to predict the reward functions from the behaviors. But how did we, the experimenters, infer the reward functions from the people? If we wrong in assumptions about which irrationalities are affecting their behavior and/or explicit reports of rewards, we would remain deluded about the subjects' true intentions and preferences.
To address these issues, we simulate demonstrator behavior subject to different irrationalities, run reward inference, and measure the performance against the ground truth, i.e., the accuracy of Bayesian posterior on the reward parameter given the (simulated) demonstrator's inputs.
2.2 BACKGROUND AND FORMALISM
Consider an FincertiinRewarte MDP (URMDP) (Bagnell et al. 2001 Regan and Boutilier 2011 Desai 2017 M (5, A, (P r,p,r), consisting of finite state and action sets and A, distributions over states {P=.a) representing the result of taking action in state discount rate E [0, 1), a (finite) set of reward parameters θ, a prior distribution E A x θ) over starting states and reward parameters, and parameterized state-action reward function θ x S x A where (0(8, a, x') represents the reward received.
R.
We assume that the human demonstrator's policy satisfies d(0), where d is an (environment- specific) planner d θ + II that returns a (possibly stochastic) policy given a particular reward parameter A. The rational demonstrator returns a policy that maximizes its expected value. On the other hand, we say that an demonstrator irrational if its planner returns policies with lower expected value than the optimal policy, for least one E θ
a planner dRational that, given reward parameter 6,
2.3 TYPES AND DEGREES OF IRRATIONALITY
There are many possible irrationalities that people exhibit (Chipman) 2014), far more than what we could study in one paper. To provide good coverage of this space, we start from the Bellman update, and systematically manipulate its terms and operators to produce a variety of different irrationalities that deviate from the optimal MDP policy in complementary ways. For instance, operating on the discount factor model myopic behavior, while operating the transition function can model optimism or the illusion of control. We parametrize each irrationality so that we can manipulate its "intensity" or deviation from rationality. Figure summarizes approach, which detail below.
3

---

Under review as a conference paper at ICLR 2021
2.3.1 RATIONAL DEMONSTRATOR
In our setup, the rational demonstrator does value iteration using the Bellman update from Fig. Our models change this update to produce different types of non-rational behavior.
2.3.2
MODIFYING THE MAX OPERATOR: BOLTZMANN
ietimmann--aatonality modifies the maximum over actions max, with Boltzmann operator with parameter B:
Vi+1()) (Bo=z)
Σ Ps.a(s') (ro(s,a,,)) +V((5)))))))))))))))))))))))))))
S'ES
where Boltz"(xx) = Σ; 2, / the most popular stochastic model used in reward CelerencelZiebart et al. 2010; Asadı and Littman 2017 Fisac et al.) 2017). After computing the value function, the Boltzmann rational planner dBouu. returns a policy where the probability of an action is proportional to the exponential of the Q Q-value of the action:
/E,ef e. (Ziebart et al. 2010 Asadi and Littman 2017 This is
(als) @Qo(s.a).
The constant 3 is called the rationality constant, because as - oc, the human choices approach perfect rationality (optimality), whereas 5 = 0 produces uniformly random choices.
2.3.3 MODIFYING THE TRANSITION FUNCTION
Our next set irrationalities manipulate the transition function away from reality.
Illusion of Control. People often overestimate their ability to control random events (Thompson 1999). To model this, we consider demonstrators that use the Bellman update:
Vi+1(s) = max Σ
P (s') (ro(s, s's + +V((s))
s'ES
where PR. (s') & (s')""))) As n co, the demonstrator acts as if it exists in deterministic environment. As n + 0, the demonstrator acts as if it had an equal chance of transitioning to every possible successor state.
Many people systematically overestimate or underestimate their chance experiencing of positive over negative events (Sharot et al. 2007).. We model this using demonstrators that modify the probability they get outcomes based on the value of those outcomes:
Optimism//Pssimisiie
V++(() max Σ (s') (ra(s, + +VVss')
e'ES
where pa (s') Ps.a (s.e('')e (1o6a)))) w controls how pessimistic or optimistic the demon- strator is. As w - to0 (respectively, > - -c),, the demonstrator becomes increasingly certain that good (bad) transitions will happen. As + 0. the demonstrator approaches the rational demonstrator.
2.3.4 MODIFYING THE REWARD: PROSPECT BIAS
Next, we consider demonstrators that use the modified Bellman update:
V++(())
max Σ Pa.(d')) (I(ro) s')) + +V((('))
E'ES
where f R is some scalar function. This equivalent solving the MDP with reward f ro- and allows to model human behavior such as loss aversion and scope insensitivity.
Prospect Bias.
Kahneman and Tversky
2013) inspires us consider a particular family of fs:
log(1 + |r])
r
fo(r)
r= 0 r r<
cc log + Irl)
c controls how loss averse the demonstrator is. As c c oc, the demonstrator primarily focuses on avoiding negative rewards. As - 0, the demonstrator focuses maximizing positive rewards.
4

---

Under review as a conference paper at ICLR 2021
2.3.5 MODIFYING THE SUM BETWEEN REWARD AND FUTURE VALUE: EXTREMAL
Extremal. People seem to exhibit duration neglect, sometimes only caring about the maximum intensity of an experience (Do et al. 2008). We model this using Bellman update:
(s) max Σ Ps.a(s') max (ro(s, a, a') (),o(s,,,s)) + aVi(s') S'ES
These demonstrators maximize the expected maximum reward along trajectory, instead of the expected sum of rewards. As a - 1, the demonstrator maximizes the expected maximum reward received along the full trajectory. As a 0, the demonstrator becomes greedy and maximizes the immediate reward.
2.3.6 MODIFYING THE DISCOUNTING
Myopic Discount. In practice, people are sometimes myopic, only considering near-term rewards. One way to model this is to decrease gamma in the Bellman update. At 7 7", the discount rate specified by the environment, this is the rational demonstrator. As 7 + 0, the demonstrator becomes greedy and only acts to maximize immediate reward.
Myopic Value Iteration. As another way to model human myopia, we consider a demonstrator that performs only h steps of Bellman updates. That is, this demonstrator for a horizon h, and discount to O reward after that. As h - oc, this demonstrator becomes rational. If h 1, this demonstrator only cares about the immediate reward.
equally about rewards
Hyperbolic Discounting. Human also exhibit hyperbolic discounting, with a high discount rate for the immediate future and a low discount rate for the far future (Grrüne-Ynnd) Yanofi 2015). Alexander and Brown (2010) formulate this as the following Bellman update:
P..('')) ro(s, s') Vi(s')
V++((s) = max
1 +VV(s's
s'ES
modulates how much the demonstrator prefers rewards now versus the future. As - 0, this demonstrator becomes a rational demonstrator without discounting.
3 EXPLORING THE EFFECTS OF KNOWN BIASES ON REWARD INFERENCE
Armed with our framework for characterizing irrationality, we test its implications for reward inference. We start by investigating the effects in random MDPs.
3.1 EXPERIMENTAL DESIGN: EXACT INFERENCE RANDOM MDPs.
Independent Variables. We manipulate the type of the planner, and vary the degree parameters for each. We use different environments, sample different ground truth reward parameters, and test different trajectory lengths for the demonstrated behavior (T 3, 15, and 30 state-actions pairs).
Dependent Measures. To separate the inference difficulty caused by suboptimal inference from the difficulty caused by demonstrator irrationality, we perform the exact Bayesian update on the trajectory& & (Ramachandran and Amir 2007). which gives us the posterior on 0 given &, P(O P(1) Jor P(OT)
Our primary metric is the expected log loss of this posterior:
Log (0s) = Eg (00())- log (())
low log loss implics that we are assigning high likelihood to the true 0*, Note that in this case, the log loss is equal to the entropy of the posterior H((lE) H(0) [((
For each environment and irrationality type, we calculate the performance of reward inference on trajectories of a fixed length T. To sample trajectory of length T from a demonstrator, fix d* and start state s. Then, we generate rollouts starting from state until T state, action pairs have been sampled from d(0*). We repeat this procedure 10 times for each start state.
5

---

Under review as a conference paper at ICLR 2021
Boltzmann
Illusion
Optimism
Prospect
3.0
3.55
3.11
2.5
2.01
10
101
10 Hyperbolic
10t
Extremal
Myopic
Myopich
2.5
Maan
5.0
0.25
0.50
0.25
1.00
0.55
0.50
0.25
1.00
2,5
5.0
7.5
100
16
R 10a
1+3
T-15
T-30
Figure 3: The log loss (lower better) of the posterior as function of the parameter we vary for each irrationality type, on the random MDP environments. For the irrationalities that interpolate to the rational planner, we denote the value that closest to rational using a dashed vertical line. Every irrationality except Prospect Bias all have parameter settings that outperform the rational planner. The error bars show the standard error of the mean, calculated by 1000 bootstraps across environments.
Simulation Environment. We used MDPs with 10 states and 2 actions, where each (s, a)-pair has We used y 0.99 and start trajectories in every state
2 random, nonzero munsition-probablliin without reward. In these 0 is a vector of length 3, where 0g is the reward received from transitions out of state i (and all other rewards are 0). We discretized each 0, with 4 values, leading | 64. We generated 20 such random MDPs.
3.2 RESULTS
Fig. plots the log loss for each irrationality for random MDPs. The degree affects reward inference, with many settings naturally resulting worse inference, especially the extremes. However, what is surprising is that every type except Prospect Bias has at least one degree (parameter) setting that results in better inference with enough data: we see that most irrationality types can be more informative than rational behavior. The more data we have, the more drastic the difference (T=30 results in both better inference and larger difference relative to rational).
We put this to the test with repeated- repeated-measures measures ANOVA with planner type the independent variable, using the data from the best parameter setting from each type and T=30, and environment as random effect. We find significant main effect for planner type (F,S, 806372) 6102.93, p < 0.001). A post-hoc analysis with Tukey HSD corrections confirmed that every irrationality except Prospect improved inference compared to the fully rational planner (p .001). For T 30, Optimism with = 3. 16 performed best, followed by Illusion of Control with n = 0.1 and Boltzmann with 3 = 10.
4 DOES THIS EFFECT GENERALIZE?
In the domain of random MDPs, we found that not only does irrationality affect reward inference. certain irrationalities can actually result better inference. In this section, we probe the generality of this finding empirically, and explain it theoretically.
4.1 GRIDWORLD
first test a toy environment that adds natural structure based on OpenAl Gym's 's "Frozen-Lake-vi) (Brockman et al.) 2016) (Fig. in the appendix). Fig. i i the appendix shows the results in the gridworld: they eerily (and reassuringly) similar to the random MDPs. This suggests that irrationalities can indeed help inference across differently structured MDPs. By inspecting the policies, we see that the irrational demonstrators were able to outperform the
Random MDPs lack structure,

---

Under review as a conference paper at ICLR 2021
Table 1:
Autonomous Driving Results: Merging
Bayesian IRL
CIOC
Horizon
Log Loss
Information Gain
Cosine Similarity
Normalized Return
3
0.690
0.696
0.999
0.939
0.824
0.562
0.940
0.981
7
1.383
0.004
0.350
0.856
rational demonstrator by disambiguating between Os that the rational demonstrator could not. To visualize this, we show examples of how the policy of several irrational demonstrators differ on the gridworld when the rational demonstrator's policies are identical in Fig. Fig. and Fig. (included in the appendix).
4.2 AUTONOMOUS DRIVING
Even if known irrationality helps reward inference both empirically and theoretically in small MDPs, the question still remains whether this result will matter in practice. Next, we investigate the effect of demonstratsor irrationality on reward inference in continuous state and action self-driving car domain (Sadigh et al. 2016). Switching to this real-world domain means we can no longer plan exactly, so we use model predictive control. also means we can no longer run exact inference, so we approximate Bayesian IRI. through samples, and also test continuous IOC (CIOC, Levine and Koltun (2012)) which recovers an approximate MLE. We measure the cosine similarity between the estimate and 0", as well as the normalized reward of the trajectory when optimizing the recovered estimate. We use the merging task from Fig. T mooee details in the appendix).
We report results in Table 1 As the columns relating Bayesian IRL show, decreasing the MPC planning horizon significantly decreases the log loss of the Bayesian IRL posterior. As with the gridworld results in section 4.1 the reason for this that the shorter planning horizons exhibit more diverse behavior (as a function of the reward). Of the 4 reward settings we used, MPC with horizon and 5 produced two different qualitative behaviors, whereas all rewards led qualitatively similar behavior with horizon 7 (depicted in Fig. D When the weight on target speed was large enough, MPC with horizon and would produce a trajectory that overtakes the front car by going off the road. In all other cases, the demonstrator would merge between the two other cars.
We found similar results for CIOC: decreasing the MPC planning horizon increases the cosine similarity between the true and CIOC-recovered rewards. The return of optimizing the CIOC- recovered rewards is also higher when the demonstrator's planning horizon is than
Overall, yet again we find that irrationality (in this case specifically myopia) improves reward inference by producing more diverse behavior as function of the reward.
4.3 THEORETICAL ANALYSIS
We now investigate this phenomenon theoretically. We show that not only can rational behavior be arbitrarily less informative than irrational behavior, but also that this applies to collzmann-raaionnalie
Informativeness mutual information. The mutual information I(0; d()) H (O)) between the policy and the reward parameters allows us to quantify the informativeness a planner. As the conditional entropy (0]d(0)) E log p( ( [d()] is equal to the log loss of the posterior under the true model, the mutual information upper bounds how much better (in terms of log loss) any inference procedure can do, relative the prior. A known insight is that planners that optimize for maximizing the discounted sum rewards are not the same as those that optimize for being informative (Dragan et al. 2013 Hadfield-Menell et al. 2016): arg max, Eg [v= * arg max, I(0; d(0)). While this insight has been tested empirically, we begin with a thcoretical lens for understanding it.
Irrationalities exist that are arbitrarily better for inference than rationality. We first consider deterministic planners: planners that return deterministic policies. We show that there cases where the rational behavior is not informative at all, whereas some (irrational) deterministic planner achieves the theoretical upper bound on informativeness, (Proof in appendix.)

---

Under review as a conference paper at ICLR 2021
True Model
Boltzamanns
14
Mean Log Logs
Loss
40
20
Rational
Optimioo
Illusion
Proppet
Eatronal
Myople 7
Myopic 8
Flyprrbdic
Heb
Hez
Random MDPs
Car Enviromeett
Figure 4: comparison of reward inference using a correct model of the irrationality type, versus always using noltzmann-raatinal model (3 10), the random MDPs (left) and the envi- ronment (right). The impairment due to model misspecification greatly outweighs the variation in inference performance caused by various irrationalities. The error bars show the standard error of the mean, calculated by the bootstrap across environments.
Proposition 1. There exists family of URMDPs with state spaces of any size, such there exists deterministic planner ditrational satisfying I(0, dirrationnt (0) -log| (+] and [0,, (R)o)ioaa(()) O.
Boltzmann-rationalit is (arbitrarily) more informative than full rationality. Of course, the upper bound is attained by some irrational planner. It demonstrates that an demonstrator can perform better than rational when specifically optimizing for informativeness. But this is artificial, contrived kind of irrationality. In fact, prior work that maximized informativeness did so by solving a more difficult problem than rationality (Dragan et al. 2013) Haddield-Mende et al.) 2016)) Here, we provide evidence that Boltzmann-rationa outperforms full rationality for reward inference. (Proof appendix.)
a standard model of stochastic choice (discussed in section 2.3
Proposition 2. There exists a family of one-state two-action MDPs, with arbitrarily large |0) such that I(0, doote = log [0] and I(0, dRational (0) = O.
5 EFFECTS OF MISSPECIFICATION ON REWARD INFERENCE
We see that irrationalities sometimes hinder, but sometimes help reward inference. So far, the learner had access to the type (and degree of irrationality) during inference. Next, we ask how important is to know this. Can we not bother with irrationality, make a default assumption, and run inference?
Assuming noisy rationality can lead to very poor inference. Fig. suggests that the answer is no. We start by comparing inference with the true model on random MDPs versus with assuming the standard Boltzmann model as default. The results are quite striking: not knowing the correct irrationality harms inference tremendously] We then confirm that misspecification greatly harms inference in the autonomous driving environment. This emphasizes the importance of understanding irrationality when doing reward inference going forward.
Approximate models of irrationality might be enough. This finding is rather daunting, as perfect models of irrationality are very challenging to develop. But do they need to be perfect? Our final analysis suggests that the answer is no as well. In Fig. of the appendix, we report the log loss of inference with the correct type, but under misspecification of the parameter. Encouragingly, find that in many cases, merely getting the type of the demonstrator irrationality correct is actually sufficient to lead much better inference than assuming Boltzmann rationality. Further, we also find evidence that the learner does not need to get the type exactly right either: as shown in Fig. [2] if the learner accounts for the demonstrator's 's myopia, but gets the type the myopia wrong, this still leads to significantly better inference than assuming Boltzmann rationality.
(Shah et al. 2019 proposed a way to model irrationality and analyzed its benefit over assuming Boltzmann; the henefit was very limited, which they attributed due to their deep learning model's brittleness compared to exact planning. Here, we compare to perfect modeling analyze the headroom that modeling has.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, page 1. ACM. 2004. URL:
eecs wew..eeeeeeeeee.....ee.eeeeeeee. wew../eley..eburrrrssslla//aaaaa s a
pdff
William H Alexander and Joshua W Brown. Hyperbolically discounted temporal difference learning.
Neural computation, 22(6):1511-1527, 2010.
Kavosh Asadi and Michael L Littman. An alternative softmax operator for reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Learning-Volume Volume 70, pages 243-252. JMLR. org. 2017.
J Andrew Bagnell, Andrew Ng, and Jeff Schneider. Solving uncertain markov decision processes. 2001.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 1oXiv:6606.01400 2016.
Susan E. F. Chipman. The Oxford Handbook of Cognitive Science. Oxford University Press, 11 2014. ISBN 9780199848 1180199442193.
Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean- Cean-Baptiste Mouret. Robots that can adapt like animals. Nature, 521(7553):503-577, 2015.
Nishant Desai. Uncertain reward-transition mdps for negotiable reinforcement learning. Technical report, UC Berkeley, Berkeley, California, USA, 2017.
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 2169-2176. IEEE, 2017.
Amy M Do, Alexander V Rupert, and George Wolford. Evaluations of pleasurable experiences: The pea rule. Psychonomic Bulletin Review, 15(1):96 98, 2008
Anca Dragan, Kenton CT Lee, and Siddhartha Srinivasa. Legibility and predictability of robot motion. In Proceedings of the 8th ACM/IEEE international conference on Human-robot interaction, pages 301-308. IEEE Press, 2013.
Owain Evans, Andreas Stuhlmüller, and Noah Goodman. Learning the prefer- ences of ignorant, inconsistent agents. In Thirtieth AAAI Conference Artifi- cial Intelligence, 2016. URL: https://pdds wttp://dd.....a.....eeeeee.e.. www.nteeeanttteeeeeeee.e wwt.aannccccooa..rgg// 31bf//
Jaime Fisac, Monica Gates, Jessica B Hamrick, Chang Liu, Dylan Hadfield- Malayandi Palaniappan, Dhruv Malik, S Shankar Sastry, Thomas L. Griffiths, and Anca D Dragan. Pragmatic- pedagogic value alignment. arXiv preprint :tti::/777.033555 2017. URL: httpt:::/ httt::htts://aaa //apxiv///
org/pdf/11/ htpppdf17001 06354.pd.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse re- inforcement learning. International Conference on Learning Representations, 2018. URL
https:///ppereeview net/ httt:/oooummmdmrktrpaaa
Till Grüne-Yanoff. Models of temporal discounting 1937-2000: interdisciplinary exchange between economics and psychology. Science in context, 28(4):675-713, 2015.
Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse
reinforcement learning. In Advances neural information processing systems, pages 3909-3917, 2016.
Antony Jameson and Eliezer Kreindler. Inverse problem of linear optimal control. SIAM Journal on Control, 1(1):1119, 1973. URL: htttps://epune siam. wttp////pp//10.111111
0311001

---

Under review as a conference paper at ICLR 2021
Mingxuan Jing. Xiaojian Ma, Wenbing Huang, Fuchun Sun, and Huaping Liu. Task transfer by preference- cost learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2471-2478, 2019.
Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. In Handbook of the fundamentals of financial decision making: Part I, pages 127. World Scientific, 2013.
Rudolf Emil Kalman. When is a linear control system optimal? Journal of Basic Engi-
neering, 86(1):51-60, 1964. URL: httppp/s/aammiiaaaaaoooeeeeeecee
asme asme org/
Eluidsengineerinn hwwpriinnngtttiiiicioo http://cee.ssttaeee/////11/3////
Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter Bentley, Samuel Bernard, Guillaume Beslon, David M Bryson, et al. The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities. arXiv preprint arXiv:/80. 803.03453, 2018.
Sergey Levine and Vladlen Koltun. Continuous inverse optimal control with locally optimal examples. arXiv preprint arXiv: 2226.666177 2012.
Katja Mombaur, Anh Truong, and Jean-Paul Laumond. From human humanoid locomotion
inverse optimal control approach. Autonomous robots, 28(3):669-383, 2010. URL: https
//link Sprinnespringg:: cww httpa///....../////.///////////.
Andrew Y Ng, Jtuart J Russell, et al. Algorithms for inverse reinorreennn learning. In /cml, volume 1,
page 2, 2000. URL: http: wttpaaataaa......aaaaaaa///////a http:////aappppeemmmeee1111-
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In I/CAI, volume pages
2007.
Sid Reddy, Anca Dragan, and Sergey Levine. Where do you think you'le re going?: Inferring beliefs about dynamics from behavior. In Advances in Neural Information Processing Systems, pages 1454-1465, 2018.
Kevin Regan and Craig Boutilier. Robust online optimization of howard-uncertann mdps. In Twenty- Second International Joint Conference on Artificial Intelligence, 2011.
Dorsa Sadigh, Shankar Sastry, Sanjit A Seshia, and Anca D) Dragan, Planning for autonomous cars that leverage effects on human actions. In Robotics: Science and Systems, volume 2. Ann Arbor, MI, USA, 2016.
Dorsa Sadigh, Anca D. Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-base based learning of reward functions. In Robotics: Science and Systems (RSS), 2017.
Rohin Shah, Noah Gundotra, Pieter Abbeel, and Anca D Dragan. On the feasibility of learning, rather than assuming, human biases for reward inference. arXiv preprint 190.v:190..0960 2019. URL:
https://arxx //rxivv org/ pwwl9//. 09624. pdd
Tali Sharot, Alison M Riceardi, Candace M Raio, and Elizabeth Phelps. Neural mechanisms mediating optimism bias. Nature, 450(7166): 2007.
Sumeet Singh, Jonathan Lacotte, Anirudha Majumdar, and Marco Pavone. Risk-sensitive inverse reinforcement learning via semi-and non-parametris methods. arXiv preprint 1711, 2017.
Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey, Journal of Machine Learning Research, 10(Jul): 1633-16663 2009.
Suzanne C Thompson. Illusions of control: How overestimate personal influence. Current Directions in Psychological Science, 16)) 1999.
Brian Ziebart, Andrew Maas, Andrew Bagnell, and Anind Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1438. Chicago, II. USA, 2008.
Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. 2010.
10

---

Under review as a conference paper at ICLR 2021
Appendices
A DETAILS FOR GRIDWORLD
In this section, we investigate the effects of irrationality on inference in an MDP based on OpenAI Gym's FrozzenLaakevo" (Brockman three types of cells: ice, holes, and rewards. The demonstrator can start in any ice cell. At each ice cell, the demonstrator can move in one of the four cardinal directions. With probability 0.8, they will go in that direction. With probability 0.2, they will instead in one of the two adjacent directions. Holes and rewards are terminal states, and return the demonstrator back to their start state. They receive penalty of 10 for falling into hole and 0; E [0, ((@) = 25) for entering into the ith reward cell.
2016). This small 5x5 gridworld (Fig. [5). consisting of
Fig. 6 shows the analogue of the results in section 3.2 for this gridworld: they reassuringly similar. We use the same experimental design as the random MDPs in section 3.1
To visualize why inference quality is improved, we show examples of how the policy of several irrational demonstrators differ on the gridworld when the rational demonstrator's policies are identical in Fig. Fig. and Fig. g
Finally, Fig. 10 example of why using the wrong model for reward inference leads to bad inference. In it, the reward inference algorithm assumes that the demonstrator in Boltzmann when it is actually Myopic. The Boltzmann rational agent would take this trajectory only the reward at the bottom not much less than the reward at the top. The myopic agent with 4, however, only "sees" the reward at the bottom. Consequently, inferring the preferences of the myopic agent as if it were Boltzmann leads to poor performance in this case.
B DETAILS FOR AUTONOMOUS DRIVING ENVIRONMENT
Even if known irrationality helps reward inference both empirically and theoretically in small MDPs, the question still remains whether this result will matter in practice. As result, we investigate the effect of demonstrator irrationality on reward inference in the self-driving car domain (Sadigh et al.) 2016).
Simulation environment. As in previous work in the car domain (Sadigh et al. 2016 2017) we model the dynamics of cars using a point-mass model. The state of each car 4-dimensional vector [xr y h v]. where x, y are the coordinates of the car, h is the heading, and e is the speed. The dimensional vector a [u1 U2]. where u1 the steering input and
control input for the car
Hille
Rewaard
Figure The gridworld used in section
11

---

Under review as a conference paper at ICLR 2021
Boltzmann
Musion
Optimism
Prospect
Mesin Log Less
2.0
10'
10'
10-1
10'
10
101
10P
10
10"
10
P
Extrémal
Myopic
Myopic
Hyperbolic
3.00
Mean Loee
2.55
225
2.00
10-1
10
1+3
T-15
Figure 6: The analog of Fig. 3 for the gridworld. Error bars are the standard error of the mean. The findings surprisingly similar with the random MDPs. Note the more limited x-axis ranges used this experiment.
4
1
1
4
Figure 7: Optimism bias (w 3.16) produces different actions for 0" = (4, 1) vs. (1, in the states shown: the rational policy is to go away from the hole regardless of 6 but optimistic demonstrator takes the chance and goes for the larger reward - up in the first case, down in the second.
U2 the acceleration. We also include a friction coefficient a. The dynamics model of the vehicle is: [ ý h [e cos h sin h uq U2
For of simulation, discretized the simulation along the time dimension using the following dynamics:
Ast = [Ax Agtt Ahe Aer] = [6, cos fy dt Tt sin by dt 0. u1 dt (u2 - a vz) dt),
where As, 8t41 and ú, + 0.5 u2 dt. In our experiments, we used di = 0.1.s.
Reward functions in this environment are assumed be a linear combination of features:
ro(s, a) T ((s,a), f S x + R
Due to environment complexity, we can't solve for optimal trajectories in our environment directly. Instead, suppose that the planner of is performing Model Predictive Control (MPC) at every
12

---

Under review as a conference paper at ICLR 2021
4
4
1
Figure 8: Myopic value iteration (h 5) produces different policies for 0* (4, 1) vs. 0" = (4. 0): while the rational expert always detours around the hole and attempts to reach the larger reward, myopia causes the myopic expert to go for the smaller source of reward when it is non-zero.
1
4
1
4
Figure 9: Boltzmann (3 100) produces different policies for 0* = (1, vs. 0" = -4, 4): when ||0|| is larger, the policy becomes closer to that of the rational demonstrator, as the differences in Q- --auuues becomes larger.
Figure 10: An example of why assuming Boltzmann is bad for myopic agent the Boltzmann rational agent would take this trajectory only if the reward at the bottom was not much less than the reward the top. The myopic agent with s 4, however, only "sees" the reward at the bottom. Consequently, inferring the preferences of the myopic agent as were Boltzmann leads to poor performance in this case.
13

---

Under review as a conference paper at ICLR 2021
iteration that is, it will plan finite horizon sequence of actions to maximize its reward, execute the first action in the sequence, then replan. As a basic model of irrationality, we consider shorter planning horizons. We assume that the reward inference procedure knows the planninng horizon exactly.
Bayesian IRL in the car environment First, we consider the analogue of the results in section 3.2 in the environment: what is the log-loss of the posterior on 0, given different planning horizons? Since the demonstrations used in this work are generated via trajectory optimization (and not an inherently stochastic process), using Bayesian IRL IRI. requires P(E
specify "fake" observation model
We use the following distribution, normal in feature space, for P(E]0):
P(E(0) (Flrtt)) (F(r(e))
where (()( is the expectation of the features of the optimal trajectory Eg for ra.
Maximum likelihood IRL in cars In practice, full Bayesian inference is completely intractable for complicated domains such as cars. Instead, the state of the art for reward inference are approximate, MLEE-
maximum-likelihod estimate (MLE) based methods. To study the effects of irrationality based inference methods, we also perform Levine and Koltun's 's Continuous IOC with Locally Optimal Examples (CIOC) (Levine and Koltun 2012). which uses a Boltzmann model of the demonstrator with second order Laplace approximation of the normalizing constant. Since our demonstrators are actually locally optimal, we rectify the bias induced by the Boltzmann model by using large B = 10a. To rectify the issue of local optimization in CIOC, we initialize the optimization procedure with the true reward weights.
As CIOC returns (approximate) MLE estimate for the reward parameter 0, we cannot use the log-loss metric. Instead, we evaluate the reward functions by the cosine similarity of 0* and Ó, as well as the normalized return of the trajectory when optimizing the recovered Ô. (We normalize the returns so that the optimal trajectory has return and the trajectory that goes forward at constant speed 0.)
Merging Our experiments were performed in a simple merging environment (Fig. D. In it, the demonstrator wants merge into the right lane while trying maintain its 1.2 forward speed. In addition to the demonstrator car, the right lane contains two constant velocity cars, traveling 0.8 speed. The features of this environment are composed of squared penalty for deviating from 1.2 forward speed, features for the squared distances to the medians of each of the lanes, feature for the minimum squared distance to any of the medians of the lanes, and a smooth collision feature.
Driving Scenario:
For the Bayesian IRL. scenario, we considered four different reward functions on this domain, consisting of varying the weight on the penalty for deviating from the target speed. All the other weights are unchanged. In particular, we considered Qspea E {0.5, 1, 2, 4]. In the CIOC scenario, we used speed = 1. We also consider 3 different planning horizons: = 3, h = 5, and h = 7. This means we had 12 conditions for Bayesian IRL and 3 for CIOC.
C PROOFS FOR SECTION
First, note that for deterministic planners, there that there are |A||s| such policies, and thus we have I(0; d())) < (d(0)) < [S] log |A| (as the entropy of discrete random variable X is bounded above by the logarithm of the size its support [X). Similarly, we also have that I(0); d())) s H()) log (0). For deterministic planners, we thus have
I(0: d(0)) min(log O0I, log [A)),
We will now prove stronger version of proposition 3.
Proposition 3. There exists a family of URMDPs with state and action spaces of any size, such that the rational planner provides no information, and there exists a deterministic planner that provides min(log 01,, [S] log (Al) bits worth of information.
14

---

Under review as a conference paper at ICLR 2021
Proof. Consider set of environments where θ 2.., -1, the prior p is uniform over θ (and thus (0) = log [9)), and where
0 a"
re(s, a, s')
0 a + a²
for some a". Then the unique optimal policy has т (s) a" for every 0 (as 0 > 0). This implies that I(0, dRational (0)) _ H (dRationaa 0.
However, the planner d' : + nt 7(0), with some fixed some ordering { (+) (i (ajis))))) of the possible policies, satisfies H ( d' (0)) = 0 and thus I(0; (0) = (0) = log (+I = |S| [A]. C
By choosing A such that |A| 2 18) / we get the Proposition 3 in the text. Proposition 4. There exists a family of one-state two-action MDPs, with arbitrarity large |0) such that I(0, dRobe(0)) = log [ and I(0, dRationa))) (0) = O.
Proof. Suppose first action d₃ has reward 0, where 0 E {1, - 1 and the second action 0g has reward O. Then the rational planner dRationl will always return a policy that always takes the first action, while dootu (0) the policy
((++)(())
To {a(a)(s))= (B(+V(()L) + (BV()))
8
1
T0(0)(81)
1 @:00
This mapping is injective, and so H (0)dRoo 0 and thus I(0; dBorz H()) log |+].
D
MORE THEORY
Why recover the reward parameter? Even irrationalities can help with inference, a natural question why we wish to infer the reward. If the actual goal is acting optimally, a rational demonstrator would be great because we can just imitate them. In addition to the fact that real people are not perfectly rational, the reason we focus on inference that imitation is not enough some cases, c.g. if we need to behave in new environments (Taylor and Stone, 2009: Devin et al. 2017 Jing et al. 2019), or transfer the policy to a robot with different dynamics (Cully et al. 2015 Reddy et al. (018)).
al. 2018
In fact, there are cases where being robust changes in dynamics requires identifying the reward parameters. Indeed, there exist such cases where it is impossible to identify the parameters from rational demonstrators, but possible from Boltzmann demonstrators:
Proposition 5. There exists URMDP M and set of new transition probabilities ((P()) (Lesa.Jeeep........ such that log [ bits of information are needed to compute the optimal policy under all the transition probabilities; and where I(0, dRational (0)) 0, whereas I(0, daatee log O
Proof. We construct a 2 state, action URMDP. Label the two states 81 and $2- Let {1, 2.., |1]. the prior be uniform over θ, and let r(s,a, $1) = 0 for all and r(s, 01, 82) = 0. r(s, a2, 52) = 1 for all s.
Under the original transition probabilities [Pa.a), aj leads deterministically to while ds deter- ministically leads to $2. Then, the unique optimal policy is ((s) = a for every f (as 0 > 0). This implies that I(9, (Rational()))
O. On the other hand, dgoltz is injective map (as in Proposition 2), and thus I(0; diBotze (0) = log ()).
Now, construct {P(()) such that aj is optimal for 0 < i 1, and ag is optimal for 8 2 i. First, let ai deterministically lead to 02, thus, leading to reward Next, let 02 lead to 81 with probability T. Therefore:
2
2i
E Bampllt (2,
2i
2i +
15

---

Under review as a conference paper at ICLR 2021
Then for A 2 i, have E p "o(ss,2,88) E pe (ro(s, 02, s')] 27+1
2 21+1
and for i we have
By construction, in order to compute the optimal policy for every transition probabilities (P (0) we must know the value of 0, and thus need log | bits of information.
E ADDITIONAL MISSPECIFICATION RESULTS
How well we have to know the planners' parameters to outperform Boltzmann-rationalite
A natural question to ask after the results of section 5 is how exactly need to know the bias before we can outperform the Ieiizmann-rationas assumption. To explore this, we compare inference with the true planner model, but with (possibly) the wrong parameter in Fig. [I In order to deal with the fact that deterministic planners can assign probability 0 to a trajectory, we smoothed the policies by applying softmax to the planners' Q-values with 8 10.0. Surprisingly, demonstrator irrationalities, wide variety of assumed parameter settings still allowed the learner to outperform inference under the Boltzmann-rational assumption. fact, for several irrationalities, the learner is always better off with the correct type, regardless of what planner parameter they assume. This suggests that we don't 't need to model human irrationalities perfectly improve on existing reward inference.
find that for many
How well do we have know the rationality type to outperform noltzmann-rationalicn
Another natural question is whether we need to know the exact type of the planner. For example, while many pieces of evidence point toward people having some form of myopia in decision making. the exact form of the myopia may not be clear. To investigate this quesiton, we compare the learner's
performance when the demonstrator assumes the wrong form Myopia in Fig. 12 As before,
we smoothed the policies by applying softmax to the planners' Q-values with B 10.0. We find that even though the learner gets the type of the irrationality wrong, the learner still does better than assuming Boltzmann- raatonnallyy.. Again, this suggests that we don't need to model human irrationalities perfectly to improve on existing reward inference.
16

---

Under review as a conference paper at ICLR 2021
Optimism
Extremal
80
Moen
100
10f
10f
10
10
10"
0.50
0.78
1,00
Prospect
Myopic
Myopic
Hyperbolic
13
20
Mean
15
10
107
0.25
0.50
0.75
2.5
7.5
10.0
10'
107
Learner assumes Bollzmaann
Leamer assumes Illusion
Leamer assumes Optimism
Learner aesrames Extremal
Prospect
Myapic
Myapic
Elyperbolic
The log loss (lower = better) of various models under parameter misspecification. Each x-axis shows the parameter that the learner assumes. The orange line represents the performance when the learner makes the faulty assumption that the demonstrator is Boltzmann-rational. In many cases, the learners perform better than by assuming Boltzmann-ratin planner correct, even if they don't get the exact parameter correct. The error bars show the standard error of the mean, calculated by the bootstrap across environments.
Figure 11:
just by getting the type the
Demonstrator has Myopic It
Demonstrator has Myopic
Mean Log Loss
15
10
5
0.2
0.4
0.6
0.8
1.0
2
10
h
Learner Bolizmann
Learner Myopic T
Learner assumes Myopic
The log loss (lower = better) of two myopic demonstrators under type misspecification. On the left, the demonstrator performs myopic value iteration (Myopic h)) but the learner assumes the demonstrator has myopic discount rate 7 (Myopic 7). On the right, the demonstrator has myopic discount rate y but the learner assumes myopic value iteration. However, in both cases, this leads to better inference than assuming Boltzmann Boltzmann-rationaaa rationality. The error bars show the standard error of the mean, calculated by the bootstrap across environments.
Figure 12:
17


Reviews:
[{"review_id": "iOMY1UgTjHg", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper investigates the effect different irrationality types have on reward inference. The irrationality types are modelled in the context of an Uncertain-Reward Markov Decision Process (URMDP) that is similar to an ordinary MDP but with a prior distribution over reward functions. Different irrationality types are expressed through different modifications of Bellman's optimality principle. In simple environment settings, given trajectories of an irrational agent with an optimal irrational policy (under the respective modified Bellman principle), Bayesian inference is feasible in order to identify a posterior over reward functions given the data (i.e. agent trajectories). \n\nThe authors provide a metric to evaluate the quality of the inference procedure. This metric is the expected log loss of the posterior evaluated at the optimal reward function (there is a further outer expectation over tasks with different reward functions). Based on this metric, the authors conclude that the quality of reward inference is higher for irrational agents than for fully rational agents, as evaluated in some grid world settings.\n\nQuality and Details\n\nI am on the verge when assessing the quality of this work. The manuscript tries to answer an interesting question and the outline and structure follow a systematic approach. However, I have 3 main concerns:\n\n1.) While there are a bunch of irrationality models presented in the context of an URMDP, it is unclear how realistic these models are for actual reward inference in practical problems.\n\n2.) I do appreciate both the empirical and theoretical analysis. However, I feel that the results may be trivial---especially in settings where an irrationality model can represent infinitely many decision-makers, one of which being a perfectly rational agent. I would intuitively argue that it is then quite \"likely\" that reward inference is easier for some of the infinitely many irrational agents as opposed to the one perfectly rational agent (recovered with a specific irrationality parameter value).\n\nSimilarly holds for the theoretical results. Proposition 1 is for example trivially true for a URMDP where the perfectly rational agent is the same for all reward functions, and where the optimal irrational agent assigns a different policy to each reward function. Proposition 2 seems to be a special case of the former for one-state-two-action settings and entropy-regularized irrationality. I simply don't know what to take away from these results...\n\n3.) The experimental setting only considers low-dimensional grid worlds (as a recommendation, I would put more emphasis on the Frozen Lake setting rather than the simpler setting). Only autonomous driving is presented as a high-dimensional task---but in this context, irrationality is only considered in terms of the agent's planning horizon but not in terms of any of the irrationality types presented earlier for lower-dimensional settings.\n\nClarity\n\nThe paper is clearly written and easy to follow.\n\nOriginality and Significance\n\nThe question, the paper asks, is original but the significance is limited. This is because it is not clear how reasonable the proposed irrationality models are, and because high-dimensional experiments with the presented irrationality models are missing.\n\nPros\n\nAn exhaustive list of irrationality models is tested.\n\nCons\n\nIt is questionable how practically relevant the irrationality models, studied by this work, are. Results may be trivial and high-dimensional experiments are largely missing.\n\n\nMinor\n\nSome more explanation in some places would help. For example, when mentioning Bayes' rule in low-dimensional settings, it could be explained more clearly how the likelihood looks like, and how the specific irrationality model affects the likelihood (through the optimal irrational policy I assume?).\n\nI would have also appreciated a bit more details about the lowest-dimensional setting: how exactly the reward parameter theta is chosen is a bit difficult to understand when reading the caption of Figure 3.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Wnn90NnmiCH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\nThis paper proposed modifications to the Bellman equation to capture known human irrationalities and showed that the reward under some conditions (the type of irrationality and parameter settings) can be better inferred compared to a rational agent. The authors demonstrated this through simulations in three different environments with different complexities and provided theoretical analyses to support this empirical finding. The authors further showed and discussed the effects on reward inference when the assumed parameter and assumed type of irrationality is misspecified. \n\n##########################################################################\n\nReasons for score: \nOverall, I vote for rejecting. The idea and results presenting are indeed very interesting and potentially promising, but I found it is to be impractical and based on lots of assumptions that easily fail. I elaborate on this below. \n \n##########################################################################\n\nPros: \n\n1. The writing is very clear and easy to follow.\n \n2.  The paper nicely applies results from studies on human behavior to AI. The idea is very interesting and can potentially inspire lots of new works in this direction. It opens up more questions than it answers. \n\n3. The authors showed the effects in three different environments, which demonstrates its generalizability to some degree. The authors also included theoretical results to support their empirical findings. \n\n##########################################################################\n\nCons: \n\n1. It seems that based on the results the author presented, we need to at least have prior knowledge about what type of irrationality humans exhibit (if not the exact parameter), this seems impractical to me and the authors did not discuss any potential approach to obtain this information. I think it is to some degree possible, as they were identified based on behavioral data empirically in human behavioral studies. \n\n2. The authors also noted that humans exhibit a mix of irrationalities in 2.1, while the papers only considered cases where there’s a single irrationality. \n\n3. In 2.1, the authors reasoned that it seems to them impossible to use actual human data, thus the paper is based on simulation. While simulations are valuable before diving into real data, I don’t think it “address these issues”. In addition, if what the authors described in 2.1 paragraph 1 are all true, it’s hard for me to see any value in this paper as it is not practical at all, i.e. the authors performed all analyses based on a miracle condition. Personally, I don’t think what authors claimed in paragraph 1 are all true, as there is a large body of experimental and computational studies in economics/psychology/neuroscience that try to answer those questions, and also noted by the authors, those irrationalities were experimentally observed. \n\n4. The formulations of various irrationalities all look reasonable to me, but it would be better if the authors can validate that those formulations actually replicate human irrationalities in the same context as in the previous literature. Otherwise, it seems the choice of formulation kind of arbitrary to me.\n\n#########################################################################\n\nOther questions/suggestions:\nIn 2.3.3 Optimism/Pessimism, the first line after the equation, should it be s’ in V_i(s) instead of s? I supposed the optimism is formulated as the agent is expected to transit into a good state with higher probability, thus it should depend on the value function of the state it transits into. \n\nI’m wondering how worse it does for reward inference if we assume rationality under different irrationalities. It would be great to add another curve in all plots in Fig. 3 to indicate the performance when ignoring irrationality. This result might validate that it is crucial to model irrationality. \n\nFor figure 4, the choice of beta = 10 seems kind of arbitrary to me. Is it the best performing parameter?\n\nWhat are the parameters used in the simulation for figure 11 (the true parameter)? It would be good to put a vertical bar to indicate the case when the true and assumed parameters are the same. I supposed it is the value when the loss is minimal. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ElExdznAsJq", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors consider the following disconnect:\n1) Bellman equation based models make the implicit assumption that the agent making the decisions is rational in a very strong sense: they know the model correctly, the discount the future exponentially, they sum the (discounted) rewards as the total reward of a policy\n2) People don't behave in this way\n\nThe disconnect between 1 and 2 is a big problem when we use inverse RL to try to learn the underlying reward functions from demonstrations since we might learn completely the wrong reward function. This can be bad for many reasons including counterfactual inference for what the person would do as well as inference for welfare (which the authors do not discuss, more on this later).\n\nGood Parts\n- The authors recognize a major problem in inverse RL from human demonstrations\n- The authors have a list of known biases and how to parametrize them in reward inference, this list is larger than what is covered in past literature\n- The authors show how badly inference can fail when the model is misspecified\n\nPlaces to Improve\n- The authors present their current contribution as more novel than it really is. \nThere is a huge literature in psychology and behavioral economics (examples: Ainslie 2001 is a whole book on the topic, O'Donoghue and Rabin 1999 American Economic Review, Laibson 1997 Quarterly Journal of Econ.) that specifically focus on what happens if someone is rational (here, discounts exponentially) but is actually irrational (discount hyperbolically). \n\nThe overall point here is both one of being able to predict behavior and one of welfare. Here, welfare refers to the following problem: if a rational agent chooses to e.g. smoke cigarettes, then clearly they prefer smoking to not smoking. On the other hand, a hyperbolic discounter may choose to smoke in the moment but also choose to take actions like throw cigarette packs away because they don't want to be tempted smoke in the future. In this case, we would infer that the smoking action was, in some sense, a mistake. Something, that we can never learn if we assume that agents are rational.\n\nIn addition, there is recent literature in the AI ethics community on precisely this same question (Peysakhovich 2019, AI Ethics and Society) that makes exactly the same point: if you do inverse RL assuming a rational model when individuals are irrational, you will learn exactly the wrong thing. \n\n- It is unclear from the paper whether the authors' current approach is doable in any real situation\nThe current setup has the issue that when doing inference we must know exactly *the way* in which the actor was irrational in order to be able to get the gains from using the correct model. However, in practice we know that agents may have one of many biases, but don't know exactly which one they have. The current theorems tell us that \"there exist MDPs\"where we can tell which irrational model is correct, however it's not immediately clear what characteristics those MDPs have and whether, for example, just by having access to the MDP and a trajectory we can say that this trajectory is consistent or inconsistent with a rational model or a single type of irrationality. \n\nThe empirical counterpart to this is Figure 4 which compares irrational actors and inference with a rational vs the correct irrational model, however since the correct irrational model isn't ever known, it seems like the better comparison would be to show rational vs. all types of irrational and see whether the correct type of irrationality is picked up.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5BuqjaGj_fS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work studies the effect of modeling systematic irrationality of the demonstrator for reward learning problems. By manipulating different factors of Bellman update, the authors simulate different irrational behavior in demonstrations. Experiments in gridworld and a 2D driving domain demonstrate that modeling irrationality helps reward inference. The authors also demonstrate that knowing the general/approximate type of irrationality, instead of knowing the exact irrationality model, might be enough to improve reward inference. \n\nPros:\nThe problem setting is important. It is significant to model human rationality and irrationality for learning from the data generated by human demonstrators; insights from studying human behavior helps construct better learning algorithms.\nThis work formalizes a diverse set of irrationality, as observed in human behavior, within the MDP formalism for systematic analysis of the effect of perfect modeling\nThe presented formalism for irrationality centers around Bellman update; such a unified presentation of the formalism for irrationality is novel, prior works only presented a few types of irrationality separately as different modifications to the value-iteration algorithm \n\nCons:\nIt is not surprising that knowing the generative bias in data helps inference; assuming access to the underlying irrationality type is a strong assumption. The more important task is to understand what types of irrationality exist in natural human data and how to infer them.\nThis work only demonstrates that (in contrast to the findings of prior work of Shah et al.) knowing the true irrationality model outperforms assuming Boltzmann-rationality by a lot, however, it does not experimentally show the effect of assuming the wrong irrationality type on learning (besides Botlzmann-rational); if inferring the type of irrationality is efficient and approximately, then it is desirable to do so whenever possible; on the other hand, if assuming the wrong type of irrationality performs worse than assuming Boltzmann-rational, then it is important to know the risk. \nThe authors also acknowledge that it is hard to infer irrationality directly from human data and real human data may contain multiple types of irrationality; it is important to discuss how this influences the contribution of this paper along with results from the above point demonstrating the effect of assuming wrong or incomplete irrationality when multiple irrationalities exist.\nIn the abstract, it is indicated that the findings of “myopic behavior being more informative” allow us to ask human demonstrators to be myopic when they demonstrate; this deviates from the motivation to correctly model human’s irrationality in the data generation process but rather circumvent the problem of inferring human’s irrationality by conditioning the demonstrator to be “myopic”. Such claims should be avoided when there are no experimental results supporting their validity.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two very confident and fairly confident reviewers rate this paper ok but not good enough, and two other fairly confident reviewers rate the article below the acceptance threshold. Therefore I must reject the article. The reviewers provided encouraging comments and suggestions on how the manuscript could be improved, which I hope the authors will find useful. 
Decision: Reject