THE IMPACTS OF KNOWN AND UNKNOWN DEMON- STRATOR IRRATIONALITY ON REWARD INFERENCE
Year: 2020

Abstract:
Algorithms inferring rewards from human behavior typically assume that people are (approximately) rational. In reality, people exhibit a wide array of irrationalities. Motivated by understanding the benefits of modeling these irrationalities, we analyze the effects that demonstrator irrationality has on reward inference. We propose operationalizing several forms of irrationality in the language of MDPs, by altering the Bellman optimality equation, and use this framework to study how these alterations affect inference. We find that incorrectly assuming noisy-rationality for an irrational demonstrator can lead to remarkably poor reward inference accuracy, even in situations where inference with the correct model leads to good inference. This suggests a need to either model irrationalities or find reward inference algorithms that are more robust to misspecification of the demonstrator model. Surprisingly, we find that if we give the learner access to the correct model of the demonstrator’s irrationality, these irrationalities can actually help reward inference. In other words, if we could choose between a world where humans were perfectly rational and the current world where humans have systematic biases, the current world might counterintuitively be preferable for reward inference. We reproduce this effect in several domains. While this finding is mainly conceptual, it is perhaps actionable as well: we might ask human demonstrators for myopic demonstrations instead of optimal ones, as they are more informative for the learner and might be easier for a human to generate.

heading: 1 INTRODUCTION
text: Motivated by difficulty in reward specification (Lehman et al., 2018), inverse reinforcement learning (IRL) methods estimate a reward function from human demonstrations (Ng et al., 2000; Abbeel and Ng, 2004; Kalman, 1964; Jameson and Kreindler, 1973; Mombaur et al., 2010). The central assumption behind these methods is that human behavior is rational, i.e., optimal with respect to their reward (cumulative, in expectation). Unfortunately, decades of research in behavioral economics and cognitive science (Chipman, 2014) have unearthed a deluge of irrationalities, i.e., of ways in which people deviate from optimal decision making: hyperbolic discounting, scope insensitivity, illusion of control, decision noise, loss aversion, to name a few.
While as a community we are starting to account for some possible irrationalities plaguing demonstrations in different ways (Ziebart et al., 2008; 2010; Singh et al., 2017; Reddy et al., 2018; Evans et al., 2016; Shah et al., 2019), we understand relatively little about what effect irrationalities have on the difficulty of inferring the reward. In this work, we seek a systematic analysis of this effect. Do irrationalities make it harder to infer the reward? Is it the case that the more irrational someone is, the harder it is to infer the reward? Do we need to account for the specific irrationality type during learning, or can we get away with the standard noisy-rationality model? The answers to these questions are important in deciding how to move forward with reward inference. If irrationality, even when well modelled, still makes reward inference very difficult, then we will need alternate ways to specify behaviors. If well-modelled irrationality leads to decent reward inference but we run into problems when we just make a noisy-rational assumption, that suggests we need to start accounting for irrationality more explicitly, or at least seek assumptions or models that are robust to many different types of biases people might present. Finally, if the noisy-rational model leads
to decent inference even when the demonstrator is irrational, then we need not dedicate significant resources to addressing irrationality.
One challenge with conducting such an analysis is that there are many irrationalities in the psychology and behavioral economics literature, with varying degrees of mathematical formalization versus empirical description. To structure the space for our analysis, we operationalize irrationalities in the language of MDPs by systematically enumerating possible deviations from the Bellman equation – imperfect maximization, deviations from the true transition function, etc. This gives us a formal framework in which we can simulate irrational behavior, run reward inference, and study its performance. Armed with this formalism, we then explore the various impacts of irrationality on reward learning in three families of environments: small random MDPs, a more legible gridworld MDP, and an autonomous driving domain drawn from the robotics literature (Sadigh et al., 2016).
Irrationality can help, rather than hinder reward inference – if it is modelled correctly. We first explore the impacts of demonstrator irrationality when the irrationality is known to the reward inference algorithm. Surprisingly, we find that certain irrationalities actually improve the quality of reward inference - that is, they make reward easier to learn. Importantly, this is not compared to assuming the wrong model of the human: our finding is that humans who exhibit (correctly modelled) irrationality are more informative than humans who exhibit (correctly modelled) rationality! This is consistent in all three domains. We explain this theoretically from the perspective of the mutual information between the demonstrator behavior and the reward parameters, proving that some irrationalities are arbitrarily more informative than rational behavior.
Unmodelled irrationality leads to remarkably poor reward inference. It might seem that we can’t immediately benefit from the knowledge that irrationalities help inference unless we have a comprehensive understanding of human decision-making, and so we should just stick to the status quo of modeling people as rational. However, we find that modeling irrational demonstrators as noisily-rational can lead to worse outcomes than not performing inference at all and just using the prior (section 5). Encouragingly, we also find evidence that even just modeling the demonstrator’s irrationality approximately allows a learner to outperform modeling the demonstrator as noisilyrational (section E).
Overall, we contribute 1) a theoretical and empirical analysis of the effects of different irrationalities on reward inference, 2) a way to systematically formalize and cover the space of irrationalities in order to conduct such an analysis, and 3) evidence for the importance and benefit of accounting for irrationality irrationality during inference.
Our results suggest that modeling people as noisily rational leads to poor reward inference, and that it is important to model the irrationalities of human demonstrators. Our good news is that if we manage to do that well, we might be better off even compared to a counterfactual world in which people are actually rational! Of course, modeling irrationality is a long term endeavour. Our near-term good news is two fold: first, irrationalities can be an ally for teaching. For example, we could ask human demonstrators to act more myopically to better communicate their reward to the learners. Second, we need not get the biases perfectly correct to do better than assuming noisy-rationality. Instead, using slightly more realistic models of human irrationality could lead to better inference.

heading: 2 FRAMEWORK: BIASES AS DEVIATIONS FROM THE BELLMAN UPDATE

heading: 2.1 EXPLORING BIASES THROUGH SIMULATION
text: While ideally we would recruit human subjects with different irrationalities and measure how well we can learn rewards, this is prohibitive because we do not get to dictate someone’s irrationality type: people exhibit a mix of them, some yet to be discovered. Further, measuring the accuracy of inference from observing real humans is complicated by the fact that we do not have ground truth access to the human’s reward function. For instance, suppose we asked subjects to produce a set of (behavior, reward function) pairs. We could then try to predict the reward functions from the behaviors. But how did we, the experimenters, infer the reward functions from the people? If we are wrong in our assumptions about which irrationalities are affecting their behavior and/or explicit reports of rewards, we would remain deluded about the subjects’ true intentions and preferences.
To address these issues, we simulate demonstrator behavior subject to different irrationalities, run reward inference, and measure the performance against the ground truth, i.e., the accuracy of a Bayesian posterior on the reward parameter given the (simulated) demonstrator’s inputs.

heading: 2.2 BACKGROUND AND FORMALISM
text: Consider an Uncertain-Reward MDP (URMDP) (Bagnell et al., 2001; Regan and Boutilier, 2011; Desai, 2017) M = (S,A, {Ps,a}, γ,Θ, p, r), consisting of finite state and action sets S and A, distributions over states {Ps,a} representing the result of taking action a in state s, discount rate γ ∈ [0, 1), a (finite) set of reward parameters Θ, a prior distribution p ∈ ∆(S×Θ) over starting states and reward parameters, and a parameterized state-action reward function r : Θ× S ×A× S → R, where rθ(s, a, s′) represents the reward received.
We assume that the human demonstrator’s policy π satisfies π = d(θ), where d is an (environmentspecific) planner d : Θ → Π that returns a (possibly stochastic) policy given a particular reward parameter θ. The rational demonstrator uses a planner dRational that, given a reward parameter θ, returns a policy that maximizes its expected value. On the other hand, we say that an demonstrator is irrational if its planner returns policies with lower expected value than the optimal policy, for at least one θ ∈ Θ.

heading: 2.3 TYPES AND DEGREES OF IRRATIONALITY
text: There are many possible irrationalities that people exhibit (Chipman, 2014), far more than what we could study in one paper. To provide good coverage of this space, we start from the Bellman update, and systematically manipulate its terms and operators to produce a variety of different irrationalities that deviate from the optimal MDP policy in complementary ways. For instance, operating on the discount factor can model myopic behavior, while operating on the transition function can model optimism or the illusion of control. We parametrize each irrationality so that we can manipulate its “intensity" or deviation from rationality. Figure 2 summarizes our approach, which we detail below.

heading: 2.3.1 RATIONAL DEMONSTRATOR
text: In our setup, the rational demonstrator does value iteration using the Bellman update from Fig. 2. Our models change this update to produce different types of non-rational behavior.

heading: 2.3.2 MODIFYING THE MAX OPERATOR: BOLTZMANN
text: Boltzmann-rationality modifies the maximum over actions maxa with a Boltzmann operator with a parameter β:
Vi+1(s) = Boltzβa ∑
s′∈S Ps,a(s
′) (rθ(s, a, s ′) + γVi(s ′)) ,
where Boltzβ(x) = ∑ i xie βxi/ ∑ i e βxi (Ziebart et al., 2010; Asadi and Littman, 2017). This is the most popular stochastic model used in reward inference(Ziebart et al., 2010; Asadi and Littman, 2017; Fisac et al., 2017). After computing the value function, the Boltzmann-rational planner dBoltz returns a policy where the probability of an action is proportional to the exponential of the Q-value of the action: π(a|s) ∝ eβQθ(s,a). The constant β is called the rationality constant, because as β →∞, the human choices approach perfect rationality (optimality), whereas β = 0 produces uniformly random choices.

heading: 2.3.3 MODIFYING THE TRANSITION FUNCTION
text: Our next set of irrationalities manipulate the transition function away from reality.
Illusion of Control. People often overestimate their ability to control random events (Thompson, 1999). To model this, we consider demonstrators that use the Bellman update:
Vi+1(s) = max a
∑ s′∈S Pns,a(s ′) (rθ(s, a, s ′) + γVi(s ′))
where Pns,a(s ′) ∝ (Ps,a(s′))n. As n → ∞, the demonstrator acts as if it exists in a deterministic environment. As n→ 0, the demonstrator acts as if it had an equal chance of transitioning to every possible successor state.
Optimism/Pessimism. Many people systematically overestimate or underestimate their chance experiencing of positive over negative events (Sharot et al., 2007). We model this using demonstrators that modify the probability they get outcomes based on the value of those outcomes:
Vi+1(s) = max a
∑ s′∈S Pωs,a(s ′) (rθ(s, a, s ′) + γVi(s ′))
where Pωs,a(s ′) ∝ Ps,a(s′)eω(rθ(s,a,s ′)+γVi(s)). ω controls how pessimistic or optimistic the demonstrator is. As ω → +∞ (respectively, ω → −∞), the demonstrator becomes increasingly certain that good (bad) transitions will happen. As ω → 0, the demonstrator approaches the rational demonstrator.

heading: 2.3.4 MODIFYING THE REWARD: PROSPECT BIAS
text: Next, we consider demonstrators that use the modified Bellman update:
Vi+1(s) = max a
∑ s′∈S Ps,a(s ′) (f(rθ(s, a, s ′)) + γVi(s ′))
where f : R→ R is some scalar function. This is equivalent to solving the MDP with reward f ◦ rθ, and allows us to model human behavior such as loss aversion and scope insensitivity.
Prospect Bias. Kahneman and Tversky (2013) inspires us to consider a particular family of fs:
fc(r) =    log(1 + |r|) r > 0 0 r = 0
−c log(1 + |r|) r < 0 c controls how loss averse the demonstrator is. As c→∞, the demonstrator primarily focuses on avoiding negative rewards. As c→ 0, the demonstrator focuses on maximizing positive rewards.

heading: 2.3.5 MODIFYING THE SUM BETWEEN REWARD AND FUTURE VALUE: EXTREMAL
text: Extremal. People seem to exhibit duration neglect, sometimes only caring about the maximum intensity of an experience (Do et al., 2008). We model this using Bellman update:
Vi+1(s) = max a
∑ s′∈S Ps,a(s ′) max (rθ(s, a, s ′), (1− α)rθ(s, a, s′) + αVi(s′))
These demonstrators maximize the expected maximum reward along a trajectory, instead of the expected sum of rewards. As α→ 1, the demonstrator maximizes the expected maximum reward received along the full trajectory. As α→ 0, the demonstrator becomes greedy and maximizes the immediate reward.

heading: 2.3.6 MODIFYING THE DISCOUNTING
text: Myopic Discount. In practice, people are sometimes myopic, only considering near-term rewards. One way to model this is to decrease gamma in the Bellman update. At γ = γ∗, the discount rate specified by the environment, this is the rational demonstrator. As γ → 0, the demonstrator becomes greedy and only acts to maximize immediate reward.
Myopic Value Iteration. As another way to model human myopia, we consider a demonstrator that performs only h steps of Bellman updates. That is, this demonstrator cares equally about rewards for a horizon h, and discount to 0 reward after that. As h→∞, this demonstrator becomes rational. If h = 1, this demonstrator only cares about the immediate reward.
Hyperbolic Discounting. Human also exhibit hyperbolic discounting, with a high discount rate for the immediate future and a low discount rate for the far future (Grüne-Yanoff, 2015). Alexander and Brown (2010) formulate this as the following Bellman update:
Vi+1(s) = max a
∑ s′∈S Ps,a(s ′) rθ(s, a, s ′) + Vi(s′) 1 + kVi(s′)s
k modulates how much the demonstrator prefers rewards now versus the future. As k → 0, this demonstrator becomes a rational demonstrator without discounting.

heading: 3 EXPLORING THE EFFECTS OF KNOWN BIASES ON REWARD INFERENCE
text: Armed with our framework for characterizing irrationality, we test its implications for reward inference. We start by investigating the effects in random MDPs.

heading: 3.1 EXPERIMENTAL DESIGN: EXACT INFERENCE IN RANDOM MDPS.
text: Independent Variables. We manipulate the type of the planner, and vary the degree parameters for each. We use different environments, sample different ground truth reward parameters, and test different trajectory lengths for the demonstrated behavior (T = 3, 15, and 30 state-actions pairs).
Dependent Measures. To separate the inference difficulty caused by suboptimal inference from the difficulty caused by demonstrator irrationality, we perform the exact Bayesian update on the trajectory ξ (Ramachandran and Amir, 2007), which gives us the posterior on θ given ξ, P (θ|ξ) = P (ξ|θ)P (θ)∫
θ′ P (ξ|θ′)P (θ′) . Our primary metric is the expected log loss of this posterior:
Log Loss(θ|ξ) = Eθ∗,ξ∼d(θ∗) [− logP (θ∗|ξ)] . A low log loss implies that we are assigning a high likelihood to the true θ∗. Note that in this case, the log loss is equal to the entropy of the posterior H(θ|ξ) = H(θ)− I(θ; ξ). For each environment and irrationality type, we calculate the performance of reward inference on trajectories of a fixed length T . To sample a trajectory of length T from a demonstrator, we fix θ∗ and start state s. Then, we generate rollouts starting from state s until T state, action pairs have been sampled from π = d(θ∗). We repeat this procedure 10 times for each start state.

heading: 3.2 RESULTS
text: Fig. 3 plots the log loss for each irrationality for random MDPs. The degree affects reward inference, with many settings naturally resulting in worse inference, especially at the extremes. However, what is surprising is that every type except Prospect Bias has at least one degree (parameter) setting that results in better inference with enough data: we see that most irrationality types can be more informative than rational behavior. The more data we have, the more drastic the difference (T=30 results in both better inference and larger difference relative to rational).
We put this to the test with a repeated-measures ANOVA with planner type as the independent variable, using the data from the best parameter setting from each type and T=30, and environment as a random effect. We find a significant main effect for planner type (F (8, 806372) = 6102.93, p < 0.001). A post-hoc analysis with Tukey HSD corrections confirmed that every irrationality except Prospect improved inference compared to the fully rational planner (p < .001). For T = 30, Optimism with ω = 3.16 performed best, followed by Illusion of Control with n = 0.1 and Boltzmann with β = 10.

heading: 4 DOES THIS EFFECT GENERALIZE?
text: In the domain of random MDPs, we found that not only does irrationality affect reward inference, certain irrationalities can actually result in better inference. In this section, we probe the generality of this finding empirically, and explain it theoretically.

heading: 4.1 GRIDWORLD
text: Random MDPs lack structure, so we first test a toy environment that adds natural structure based on OpenAI Gym’s ‘Frozen-Lake-v0’ (Brockman et al., 2016) (Fig. 5 in the appendix). Fig. 6 in the appendix shows the results in the gridworld: they are eerily (and reassuringly) similar to the random MDPs. This suggests that irrationalities can indeed help inference across differently structured MDPs. By inspecting the policies, we see that the irrational demonstrators were able to outperform the
rational demonstrator by disambiguating between θs that the rational demonstrator could not. To visualize this, we show examples of how the policy of several irrational demonstrators differ on the gridworld when the rational demonstrator’s policies are identical in Fig. 7, Fig. 8, and Fig. 9 (included in the appendix).

heading: 4.2 AUTONOMOUS DRIVING
text: Even if known irrationality helps reward inference both empirically and theoretically in small MDPs, the question still remains whether this result will matter in practice. Next, we investigate the effect of demonstratsor irrationality on reward inference in a continuous state and action self-driving car domain (Sadigh et al., 2016). Switching to this real-world domain means we can no longer plan exactly, so we use model predictive control. It also means we can no longer run exact inference, so we approximate Bayesian IRL through samples, and also test continuous IOC (CIOC, Levine and Koltun (2012)) which recovers an approximate MLE. We measure the cosine similarity between the estimate and θ∗, as well as the normalized reward of the trajectory when optimizing the recovered estimate. We use the merging task from Fig. 1 (more details in the appendix).
We report our results in Table 1. As the columns relating to Bayesian IRL show, decreasing the MPC planning horizon significantly decreases the log loss of the Bayesian IRL posterior. As with the gridworld results in section 4.1, the reason for this is that the shorter planning horizons exhibit more diverse behavior (as a function of the reward). Of the 4 reward settings we used, MPC with horizon 3 and 5 produced two different qualitative behaviors, whereas all rewards led to qualitatively similar behavior with horizon 7 (depicted in Fig. 1). When the weight on target speed was large enough, MPC with horizon 3 and 5 would produce a trajectory that overtakes the front car by going off the road. In all other cases, the demonstrator would merge between the two other cars.
We found similar results for CIOC: decreasing the MPC planning horizon increases the cosine similarity between the true and CIOC-recovered rewards. The return of optimizing the CIOCrecovered rewards is also higher when the demonstrator’s planning horizon is 3 or 5 than 7.
Overall, yet again we find that irrationality (in this case specifically myopia) improves reward inference by producing more diverse behavior as a function of the reward.

heading: 4.3 THEORETICAL ANALYSIS
text: We now investigate this phenomenon theoretically. We show that not only can rational behavior be arbitrarily less informative than irrational behavior, but also that this applies to Boltzmann-rationality.
Informativeness as mutual information. The mutual information I(θ; d(θ)) = H(θ)−H(θ|d(θ)) between the policy and the reward parameters allows us to quantify the informativeness of a planner. As the conditional entropy H(θ|d(θ)) = E [− log p(θ|d(θ)] is equal to the log loss of the posterior under the true model, the mutual information upper bounds how much better (in terms of log loss) any inference procedure can do, relative to the prior. A known insight is that planners that optimize for maximizing the discounted sum of rewards are not the same as those that optimize for being informative (Dragan et al., 2013; Hadfield-Menell et al., 2016): arg maxdEθ[V d(θ) θ ] 6= arg maxd I(θ; d(θ)). While this insight has been tested empirically, we begin with a theoretical lens for understanding it.
Irrationalities exist that are arbitrarily better for inference than rationality. We first consider deterministic planners: planners that return deterministic policies. We show that there are cases where the rational behavior is not informative at all, whereas some (irrational) deterministic planner achieves the theoretical upper bound on informativeness. (Proof in appendix.)
Proposition 1. There exists a family of URMDPs with state spaces of any size, such there exists a deterministic planner dIrrational satisfying I(θ, dIrrational(θ)) = log |Θ| and I(θ, dRational(θ)) = 0.
Boltzmann-rationality is (arbitrarily) more informative than full rationality. Of course, the upper bound is attained by some irrational planner. It demonstrates that an demonstrator can perform better than rational when specifically optimizing for informativeness. But this is an artificial, contrived kind of irrationality. In fact, prior work that maximized informativeness did so by solving a more difficult problem than rationality (Dragan et al., 2013; Hadfield-Menell et al., 2016). Here, we provide evidence that Boltzmann-rationality, a standard model of stochastic choice (discussed in section 2.3), outperforms full rationality for reward inference. (Proof in appendix.) Proposition 2. There exists a family of one-state two-action MDPs, with arbitrarily large |Θ| such that I(θ, dBoltz(θ)) = log |Θ| and I(θ, dRational(θ)) = 0.

heading: 5 EFFECTS OF MISSPECIFICATION ON REWARD INFERENCE
text: We see that irrationalities sometimes hinder, but sometimes help reward inference. So far, the learner had access to the type (and degree of irrationality) during inference. Next, we ask how important it is to know this. Can we not bother with irrationality, make a default assumption, and run inference?
Assuming noisy rationality can lead to very poor inference. Fig. 4 suggests that the answer is no. We start by comparing inference with the true model on random MDPs versus with assuming the standard Boltzmann model as a default. The results are quite striking: not knowing the correct irrationality harms inference tremendously.1 We then confirm that misspecification greatly harms inference in the autonomous driving environment. This emphasizes the importance of understanding irrationality when doing reward inference going forward.
Approximate models of irrationality might be enough. This finding is rather daunting, as perfect models of irrationality are very challenging to develop. But do they need to be perfect? Our final analysis suggests that the answer is no as well. In Fig. 11 of the appendix, we report the log loss of inference with the correct type, but under misspecification of the parameter. Encouragingly, we find that in many cases, merely getting the type of the demonstrator irrationality correct is actually sufficient to lead to much better inference than assuming Boltzmann rationality. Further, we also find evidence that the learner does not need to get the type exactly right either: as shown in Fig. 12, if the learner accounts for the demonstrator’s myopia, but gets the type of the myopia wrong, this still leads to significantly better inference than assuming Boltzmann rationality.
1(Shah et al., 2019) proposed a way to model irrationality and analyzed its benefit over assuming Boltzmann; the benefit was very limited, which they attributed due to their deep learning model’s brittleness compared to exact planning. Here, we compare to perfect modeling to analyze the headroom that modeling has.

heading: A DETAILS FOR GRIDWORLD
text: In this section, we investigate the effects of irrationality on inference in an MDP based on OpenAI Gym’s ‘Frozen-Lake-v0’ (Brockman et al., 2016). This a small 5x5 gridworld (Fig. 5), consisting of three types of cells: ice, holes, and rewards. The demonstrator can start in any ice cell. At each ice cell, the demonstrator can move in one of the four cardinal directions. With probability 0.8, they will go in that direction. With probability 0.2, they will instead go in one of the two adjacent directions. Holes and rewards are terminal states, and return the demonstrator back to their start state. They receive a penalty of −10 for falling into a hole and θi ∈ [0, 4] (|Θ| = 25) for entering into the ith reward cell.
Fig. 6 shows the analogue of the results in section 3.2 for this gridworld: they are reassuringly similar. We use the same experimental design as the random MDPs in section 3.1.
To visualize why inference quality is improved, we show examples of how the policy of several irrational demonstrators differ on the gridworld when the rational demonstrator’s policies are identical in Fig. 7, Fig. 8, and Fig. 9.
Finally, Fig. 10 example of why using the wrong model for reward inference leads to bad inference. In it, the reward inference algorithm assumes that the demonstrator in Boltzmann when it is actually Myopic. The Boltzmann rational agent would take this trajectory only if the reward at the bottom was not much less than the reward at the top. The myopic agent with n ≤ 4, however, only "sees" the reward at the bottom. Consequently, inferring the preferences of the myopic agent as if it were Boltzmann leads to poor performance in this case.

heading: B DETAILS FOR AUTONOMOUS DRIVING ENVIRONMENT
text: Even if known irrationality helps reward inference both empirically and theoretically in small MDPs, the question still remains whether this result will matter in practice. As a result, we investigate the effect of demonstrator irrationality on reward inference in the self-driving car domain (Sadigh et al., 2016).
Simulation environment. As in previous work in the car domain (Sadigh et al., 2016; 2017), we model the dynamics of cars using a point-mass model. The state of each car is a 4-dimensional vector s = [x y h v], where x, y are the coordinates of the car, h is the heading, and v is the speed. The control input for the car is a two dimensional vector a = [u1 u2], where u1 is the steering input and
u2 is the acceleration. We also include a friction coefficient α. The dynamics model of the vehicle is:
[ẋ ẏ ḣ v̇] = [v · cosh v · sinh v · u1 u2 − α · v].
For ease of simulation, we discretized the simulation along the time dimension using the following dynamics:
∆st = [∆xt ∆yt ∆ht ∆vt] = [v̄t · cosht · dt v̄t · sinht · dt v̄t · u1 · dt (u2 − α · vt) · dt], where ∆st = st+1 − st and v̄t = vt + 0.5 · u2 · dt. In our experiments, we used dt = 0.1s. Reward functions in this environment are assumed to be a linear combination of features:
rθ(s, a) = θ >f(s, a), f : S ×A→ Rn
Due to environment complexity, we can’t solve for optimal trajectories in our environment directly. Instead, we suppose that the planner of is performing Model Predictive Control (MPC) at every
iteration – that is, it will plan a finite horizon sequence of actions to maximize its reward, execute the first action in the sequence, then replan. As a basic model of irrationality, we consider shorter planning horizons. We assume that the reward inference procedure knows the planninng horizon exactly.
Bayesian IRL in the car environment First, we consider the analogue of the results in section 3.2 in the car environment: what is the log-loss of the posterior on θ, given different planning horizons? Since the demonstrations used in this work are generated via trajectory optimization (and not an inherently stochastic process), using Bayesian IRL requires us to specify a “fake" observation model P (ξ|θ). We use the following distribution, normal in feature space, for P (ξ|θ):
P (ξ|θ) ∝ e(E[f(ξ)]−E[f(ξ ∗ θ ]) TΣ−1 ξ∗ θ (E[f(ξ)]−E[f(ξ∗θ ]),
where E[f(ξ∗θ ] is the expectation of the features of the optimal trajectory ξ ∗ θ for rθ.
Maximum likelihood IRL in cars In practice, full Bayesian inference is completely intractable for complicated domains such as cars. Instead, the state of the art for reward inference are approximate, maximum-likelihood estimate (MLE) based methods.To study the effects of irrationality on MLEbased inference methods, we also perform Levine and Koltun’s Continuous IOC with Locally Optimal Examples (CIOC) (Levine and Koltun, 2012), which uses a Boltzmann model of the demonstrator with a second order Laplace approximation of the normalizing constant. Since our demonstrators are actually locally optimal, we rectify the bias induced by the Boltzmann model by using a large β = 104. To rectify the issue of local optimization in CIOC, we initialize the optimization procedure with the true reward weights.
As CIOC returns an (approximate) MLE estimate for the reward parameter θ, we cannot use the log-loss metric. Instead, we evaluate the reward functions by the cosine similarity of θ∗ and θ̂, as well as the normalized return of the trajectory when optimizing the recovered θ̂. (We normalize the returns so that the optimal trajectory has return 1 and the trajectory that goes forward at constant speed 0.)
Driving Scenario: Merging Our experiments were performed in a simple merging environment (Fig. 1). In it, the demonstrator wants to merge into the right lane while trying to maintain its 1.2 forward speed. In addition to the demonstrator car, the right lane contains two constant velocity cars, traveling at 0.8 speed. The features of this environment are composed of a squared penalty for deviating from 1.2 forward speed, features for the squared distances to the medians of each of the lanes, a feature for the minimum squared distance to any of the medians of the lanes, and a smooth collision feature.
For the Bayesian IRL scenario, we considered four different reward functions on this domain, consisting of varying the weight on the penalty for deviating from the target speed. All the other weights are unchanged. In particular, we considered θspeed ∈ {0.5, 1, 2, 4}. In the CIOC scenario, we used θspeed = 1. We also consider 3 different planning horizons: h = 3, h = 5, and h = 7. This means we had 12 conditions for Bayesian IRL and 3 for CIOC.

heading: C PROOFS FOR SECTION 4.3
text: First, note that for deterministic planners, there that there are |A||S| such policies, and thus we have I(θ; d(θ)) ≤ H(d(θ)) ≤ |S| log |A| (as the entropy of a discrete random variable X is bounded above by the logarithm of the size of its support |X|). Similarly, we also have that I(θ; d(θ)) ≤ H(θ) ≤ log |Θ|. For deterministic planners, we thus have
I(θ; d(θ)) ≤ min(log |Θ|, |S| log |A|),
We will now prove a stronger version of proposition 3.
Proposition 3. There exists a family of URMDPs with state and action spaces of any size, such that the rational planner provides no information, and there exists a deterministic planner that provides min(log |Θ|, |S| log |A|) bits worth of information.
Proof. Consider a set of environments where Θ = {1, 2, ..., |A||S|}, the prior p is uniform over Θ (and thus H(θ) = log |Θ|), and where
rθ(s, a, s ′) =
{ θ a = a∗
0 a 6= a∗
for some a∗. Then the unique optimal policy has π(s) = a∗ for every θ (as θ > 0). This implies that
I(θ, dRational(θ)) ≤ H(dRational(θ)) = 0.
However, the planner d′ : θ 7→ π(θ), with some fixed some ordering {π(i)}i∈{1,2,...,|A||S|} of the possible policies, satisfiesH(θ|d′(θ)) = 0 and thus I(θ; d′(θ)) = H(θ) = log |Θ| = |S| log |A|.
By choosing A such that |A| ≥ |θ|1/|S|, we get the Proposition 3 in the text. Proposition 4. There exists a family of one-state two-action MDPs, with arbitrarily large |Θ| such that I(θ, dBoltz(θ)) = log |Θ| and I(θ, dRational(θ)) = 0.
Proof. Suppose first action a1 has reward θ, where θ ∈ {1, 2, ...,−|Θ|}, and the second action a2 has reward 0. Then the rational planner dRational will always return a policy that always takes the first action, while dBoltz(θ) is the policy
πθ(a1|s1) = eβ(θ+V (s1))
eβ(θ+V (s1)) + eβV (s1) =
1
1 + e−βθ
πθ(a2|s1) = 1
1 + eβθ
This mapping is injective, and so H(θ|dBoltz(θ)) = 0 and thus I(θ; dBoltz(θ)) = H(θ) = log |Θ|.

heading: D MORE THEORY
text: Why recover the reward parameter? Even if irrationalities can help with inference, a natural question why we wish to infer the reward. If the actual goal is acting optimally, a rational demonstrator would be great because we can just imitate them. In addition to the fact that real people are not perfectly rational, the reason we focus on inference is that imitation is not enough in some cases, e.g. if we need to behave in new environments (Taylor and Stone, 2009; Devin et al., 2017; Jing et al., 2019), or transfer the policy to a robot with different dynamics (Cully et al., 2015; Fu et al., 2018; Reddy et al., 2018).
In fact, there are cases where being robust to changes in dynamics requires identifying the reward parameters. Indeed, there exist such cases where it is impossible to identify the parameters from rational demonstrators, but possible from Boltzmann demonstrators: Proposition 5. There exists an URMDP M and a set of new transition probabilities {{P (i)s,a}}i∈{1,2,...,|Θ|} such that log |Θ| bits of information are needed to compute the optimal policy under all the transition probabilities; and where I(θ, dRational(θ)) = 0, whereas I(θ, dBoltz(θ)) = log |Θ|.
Proof. We construct a 2 state, 2 action URMDP. Label the two states s1 and s2. Let Θ = {1, 2, ..., |Θ|}, the prior be uniform over Θ, and let r(s, a, s1) = θ for all s, a and r(s, a1, s2) = 0, r(s, a2, s2) = −1 for all s. Under the original transition probabilities {Ps,a}, a1 leads deterministically to s1 while a2 deterministically leads to s2. Then, the unique optimal policy is π(s) = a∗ for every θ (as θ > 0). This implies that I(θ, dRational(θ)) = 0. On the other hand, dBoltz is an injective map (as in Proposition 2), and thus I(θ; dBoltz(θ)) = log |Θ|. Now, we construct {P (i)s,a} such that a1 is optimal for θ ≤ i− 1, and a2 is optimal for θ ≥ i. First, let a1 deterministically lead to a2, thus, leading to reward 0. Next, let a2 lead to s1 with probability
2 2i+1 . Therefore:
E s′∼P (i)s,a2
[rθ(s, a2, s ′)] =
2 2i+ 1 θ − 2i− 1 2i+ 1
Then for θ ≥ i, we have E s′∼P (i)s,a2 [rθ(s, a2, s ′)] ≥ 12i+1 > 0 and for θ ≤ i − 1, we have E s′∼P (i)s,a2 [rθ(s, a2, s ′)] ≤ − 12i+1 < 0. By construction, in order to compute the optimal policy for every new transition probabilities {P (i)s,a}, we must know the value of θ, and thus need log |Θ| bits of information.

heading: E ADDITIONAL MISSPECIFICATION RESULTS
text: How well do we have to know the planners’ parameters to outperform Boltzmann-rationality?
A natural question to ask after the results of section 5 is how exactly we need to know the bias before we can outperform the Boltzmann-rational assumption. To explore this, we compare inference with the true planner model, but with (possibly) the wrong parameter in Fig. 11. In order to deal with the fact that deterministic planners can assign probability 0 to a trajectory, we smoothed the policies by applying a softmax to the planners’ Q-values with β = 10.0. Surprisingly, we find that for many demonstrator irrationalities, a wide variety of assumed parameter settings still allowed the learner to outperform inference under the Boltzmann-rational assumption. In fact, for several irrationalities, the learner is always better off with the correct type, regardless of what planner parameter they assume. This suggests that we don’t need to model human irrationalities perfectly to improve on existing reward inference.
How well do we have to know the rationality type to outperform Boltzmann-rationality?
Another natural question is whether we need to know the exact type of the planner. For example, while many pieces of evidence point toward people having some form of myopia in decision making, the exact form of the myopia may not be clear. To investigate this quesiton, we compare the learner’s performance when the demonstrator assumes the wrong form of Myopia in Fig. 12. As before, we smoothed the policies by applying a softmax to the planners’ Q-values with β = 10.0. We find that even though the learner gets the type of the irrationality wrong, the learner still does better than assuming Boltzmann-rationality. Again, this suggests that we don’t need to model human irrationalities perfectly to improve on existing reward inference.

Review summaries:
This paper investigates the effect different irrationality types have on reward inference. The irrationality types are modelled in the context of an Uncertain-Reward Markov Decision Process (URMDP) that is similar to an ordinary MDP but with a prior distribution over reward functions. Different irrationality types are expressed through different modifications of Bellman's optimality principle. In simple environment settings, given trajectories of an irrational agent with an optimal irrational policy (under the respective modified Bellman principle), Bayesian inference is feasible in order to identify a posterior over reward functions given the data (i.e. agent trajectories). 

This work studies the effect of modeling systematic irrationality of the demonstrator for reward learning problems. By manipulating different factors of Bellman update, the authors simulate different irrational behavior in demonstrations. Experiments in gridworld and a 2D driving domain demonstrate that modeling irrationality helps reward inference. The authors also demonstrate that knowing the general/approximate type of irrationality, instead of knowing the exact irrationality model, might be enough to improve reward inference. 

This paper proposed modifications to the Bellman equation to capture known human irrationalities and showed that the reward under some conditions (the type of irrationality and parameter settings) can be better inferred compared to a rational agent. The authors demonstrated this through simulations in three different environments with different complexities and provided theoretical analyses to support this empirical finding. The authors further showed and discussed the effects on reward inference when the assumed parameter and assumed type of irrationality is misspecified.